Project: Incremental Improvements to Debian’s CI platform
Project Link: https://summerofcode.withgoogle.com/projects/#6433686825730048
Code Repository: https://salsa.debian.org/ci-team/debci
Mentors: Antonio Terceiro and Paul Gevers

About the Project

Debian Continuous Integration is Debian’s CI platform. It runs tests on the packages published in the Debian archive, and today is used to control migration of packages from unstable, Debian’s development area, to testing, the area of the archive where the next Debian release is being prepared. This makes it a crucial part of Debian’s infrastructure.

The web platform shows the results of all the tests executed. Debian CI provides developers both API and a GUI Self-Service section to request tests for the packages and get information on test history.

This project involves implementing incremental improvements to the platform, making it easier to use and maintain.

Deliverables of the project:

• Migrating Logins to Salsa
• Adding support for testing security uploads and Debian LTS

Work done

Migrating Logins to Salsa

• Originally, Debian CI used Debian SSO client certificates for logging in, but since that was deprecated logins are migrated to Salsa, Debian’s Gitlab instance and this is implemented with the help of OmniAuth, the ruby authentication framework.

• Another thing fixed as part of this task was that there exists a limitation in the existing database structure, where usernames were directly stored as the test requestor field, so that relationship was normalized with a proper foreign key to the users table.

Merge Requests:

• Migrate to logins using Salsa
• Update the jobs table to add real reference to users table

Adding support for testing security uploads and Debian LTS

In this task, work was done to enable private tests in Debian CI for adding support for testing security uploads and Debian LTS. It includes tasks from adding the private field in the API and Self-Service section in requesting tests to adding an option to publish them when ready through both interfaces.

Merge Requests:

• Add private jobs flag to jobs
• Submit private jobs via API
• Fix in migration file
• Submit private jobs via Self Service
• Add extra-apt-sources in API
• Add extra-apt-sources via Self Service
• Validate extra apt sources
• Publish private tests via API
• Update Self-Service Job History Page
• Add Publish and Retry in Self-Service

Others

I worked on some issues to add usability improvements for the web interface:

• Change Retry link to point to the self-service interface
• Add a user menu in the self-service interface

Learning Takeaways

I learned a lot throughout the entire journey. Some of the things to mention are:

• Writing tests: I learned about using Rspec for writing tests in ruby and understood how writing tests is an integral part of coding any project.
• Using good coding practices: By my merge requests reviews done by my mentor Antonio, I came to know about good coding practices which help in keeping the code both clean and concise.

Acknowledgement

I owe huge thanks to my mentors Antonio Terceiro and Paul Gevers for their constant support and guidance throughout the entire duration. It is for them that I was able to get started with the code-base of the project in the first place. And while working on the project, they were extremely responsive to all my queries. My merge requests were all thoroughly reviewed which enables me to learn more and work more efficiently. It was a complete pleasure interacting with them on weekly meet calls.

To sum up, my GSoC journey was awesome 🎉 and I had a fun and productive summer with Debian 😃

What’s next?

As part of a continuation of my project task Adding support for testing security uploads and Debian LTS, I would be working on Debian CI to facilitate the process of connecting the embargoed archive.

It’s the end of my GSoC journey but certainly not the end of my journey with Debian or Open-Source. I plan to stay an active contributor to Debian and get involve with more Open Source projects as well 🌟

Extras

The link to my Salsa Profile for all my work activities can be found here.
To know more about my journey, my other GSoC blog posts can be found below:

• Journey to GSoC
• GSoC: About my Project and Community Bonding Period
• GSoC: First Phase of Coding Period
• GSoC: Second Phase of Coding Period

Hello there.

So here we are near the end of GSoC 2021 and with that, I am sharing details of the work I completed in the second phase of the coding period.

Continuing with details of the task I left on…

Task: Adding support for testing security uploads and Debian LTS

• The extra-apt-sources parameter is added to both API and Self-Service section for the test request object and the part which took some time was deciding on its validation since if you ever deal with apt sources, you would know that there exist a lot of combinations of a valid apt source and in the end, the decision was taken to just add the minimal checks.

• Since, we have enough setup to request private tests, it made sense to have a way of looking them up in the Self-Service section. So a new column is added to the test records in Job-History marking the visibility of a test (private or public) and a new filter to filter out records based on the visibility.

• The option to retry tests is also added in Self-Service in the Job-History section so that private tests can also be retried. As earlier, this option was only available in Package History pages but those pages do not show the private tests.

• The next thing in the list was to publish these test results of private tests at some point. For the API section, a new endpoint is added to accept a string of run_ids which is ready to be published, and for the Self-Service , in the Job-History page a Publish button is added to be clicked after check boxing the required tests.

This ends the list of my planned work for the project. 🎉

Some more interesting stuff

As we got some more time for the coding period to end, my mentor Antonio Terceiro suggested that I could pick any issue from the opened issues list to work on so I picked up the one to add a user menu for the Self-Service section and completed that.

Antonio also gave me insights into Debian Packaging and even let me try packaging a newer upstream version of the package itamae which is one of the packages maintained by him.

I also worked on creating a video about my GSoC project for DebConf21 and this was truly the hardest thing to do. From setting up everything to final editing, it took me about 2 days to create something sane to submit. Now, I am excited about DebConf21 😃

That concludes my work in the second phase of the coding period and next, I will be sharing my Final Evaluation Submission real soon.

Stay tuned!

Hello there.

I still can’t believe that the first half of GSoC period is almost over. So it’s been about 5 weeks working on the project and that means I have a lot to share about it. So without further ado, let’s get started.

I will be listing up my work done in the respective tasks.

Task: Migrating Logins to Salsa

The objective of this task was that the users could log in to their account on debci using their Debian Salsa account (collaborative development server for Debian based on the GitLab software) and this is implemented with the help of OmniAuth, the ruby authentication framework.

At the beginning of this, I had to discuss quite a few issues with my mentors that I was bumping into, and by the end of it with multiple revisions and discussions, the following was implemented:

• The previous users' table schema of debci comprises the username field which contained mostly the emails of the users with some exceptions and to accommodate the Salsa logins, a new uid field is added to the table to store the Salsa uid of the logged-in user with the username field storing Salsa usernames now and as the Salsa users have the liberty to change their usernames, the updation of username as well as in debci database is also taken care of.

• For Salsa login, the ruby-omniauth-gitlab strategy has been used and for login in development mode, the developer strategy which comes with ruby-omniauth has been set up.

• Added a Login Page giving the option to log in using Salsa and an additional option to login in Developer Mode which is accessible only in Development Setup so that other contributors don’t have to set up dummy Salsa applications for working.

• Added specs for the new login process. This was an interesting part, as I got the chance to understand RSpec and facilities provided by OmniAuth to mock the authentication for Integration Testing.

• One blocker that I dealt with was that the Debian release from where packages were pulled out for debci have the OmniAuth version 1.8, which was not working well with the developer strategy implementation for the application so to resolve that I did a minor change to the callback API for developer strategy until the time that release have the newer version of OmniAuth.

• Another thing we discussed in one of the meetings that in the existing database structure, the tests do not have a real reference to the users' table and rather the username is stored directly as a string for the requestor field, so this thing was fixed as part of this task.

The migration of the existing users' data for the new logins was handled by my mentor Antonio Terceiro and with this, our first task is concluded. All these changes are now part of Debian Continuous Integration platform and you can find the blogpost for same by Antonio here.

This task also allowed me to write my first ever tutorial Tutorial: Integrating OmniAuth with Sinatra Application to help people looking to integrate their ruby application with OmniAuth.

Moving further to the next task in progress.

Task: Adding support for testing security uploads and Debian LTS

This is the next task I am working on enabling private tests in debci for adding support for testing security uploads and Debian LTS. Since it’s a bigger task, it is broken down into about 6-7 steps and till now, the following has been done:

• The schema of jobs' (tests) table is updated to have a boolean field to store whether the job is private or not.

• The is_private parameter is added to both API and Self-Service section so the private test can be submitted through the API as well as through GUI form on the web platform.

• Another thing which comes up through discussion in meetings that a parameter is required to add extra-apt-sources for getting packages of security repository and this is the part in progress.

So that concludes my work till now. It has been an amazing journey with lots of learning and also the guidance from the wonderful mentors of my project and I am looking forward to more exciting parts ahead.

That’s all for now. See you next time!

As part of my GSoC project, my first task includes that user could login into their account on debci using their Debian Salsa account (collaborative development server for Debian based on the GitLab software).

The task is officially completed using OmniAuth library and while implementing it, I found that the documentation of OmniAuth is quite a mix-match and more focused on using it with Rails app and this gives me a idea to write a tutorial for people looking to integrate OmniAuth with Sinatra application. So here it is.

Now, depending on the provider, Omniauth requires specific strategy which are generally released individually as RubyGems. For this tutorial, I would be using omniauth-gitlab which I used for Debian Salsa in my project, omniauth-twitter and a developer strategy which could be used for project in development mode and comes with omniauth gem itself.

For simplicity purposes, I have included all routes and OmniAuth configurations in a single file app.rb.

Let’s start.

Register your application

This can be easily done – just head over to provider (Twitter, Salsa) website and find the option to create a new application and fill in the form. In the callback URL field, you need to append /auth/:provider/callback to whichever URL you used in the website field.

The client-id and client-secret is obtained from the console which is used in further step to set up OmniAuth.

Gems

At the top of the file, we require the necessary gems of for our project

require 'sinatra'
require 'omniauth'
require 'omniauth-gitlab'
require 'omniauth-twitter'

Enable sessions

In order for OmniAuth to work and to store the logged in user across requests, sessions need to be activated and if activated, you have one session hash per user session.

configure do
    set :sessions, true
end

Set up OmniAuth configurations

OmniAuth::Builder Rack middleware build up your list of OmniAuth strategies for use in your application:

use OmniAuth::Builder do
    if development?
      provider :developer,
               fields: [:name],
               uid_field: :name
    end
    provider :gitlab, #client-id , #client-secret,
             scope: "read_user",
             client_options: {
               site: 'https://salsa.debian.org/api/v4/'
             }
    provider :twitter, #client-id , #client-secret,
end

Here, few things to note could be the extra options used with providers, as
In developer:
fields : to specify the form fields for login in developer mode and by default it has name and email.
uid_field: to specify that which field’s value could be obtained as uid and by default it is email.
In gitlab:
scope: to limit the scope of application, by default, the api scope is requested and must be allowed in GitLab’s application configuration.
client_options: to specify the server being used as client based on Gitlab software.
Note: In case you want to have a different callback url other than the default /auth/:provider/callback, it can be specified using redirect_url option in case of gitlab provider and accordingly update it in your Application Configuration at provider’s console.

Extra configurations

To redirect to auth/failure route in case of failure even in developer mode, following could be added:

OmniAuth.config.on_failure = proc do |env|
    OmniAuth::FailureEndpoint.new(env).redirect_to_failure
end

By default, OmniAuth will log to STDOUT but you can configure this. If you don’t want OmniAuth to log to STDOUT, following could be used:

OmniAuth.config.logger.level = Logger::UNKNOWN

Setting up routes

Login route

Starting with GET /login route, where you can specify the options available to login:

get '/login' do
    <<~HTML
    <form method='post' action='/auth/gitlab'>
    <input type="hidden" name="authenticity_token" value='#{request.env["rack.session"]["csrf"]}'>
    <button type='submit'>Login with Salsa</button>
    </form>
    <form method='post' action='/auth/twitter'>
    <input type="hidden" name="authenticity_token" value='#{request.env["rack.session"]["csrf"]}'>
    <button type='submit'>Login with Twitter</button>
    </form>
    <form method='post' action='/auth/developer'>
    <input type="hidden" name="authenticity_token" value='#{request.env["rack.session"]["csrf"]}'>
    <button type='submit'>Login with Developer</button>
    </form>
  HTML
end

The auth/:provider path is created and configured automatically by OmniAuth, so you just need to send the request to that paths and auth process will start.
As per the OmniAuth version 2.0, OmniAuth now defaults to only POST as allowed request_phase methods and authenticity_token is required to validate your requests so make sure to take care of this.

Callback routes:

On success from authentication, Omniauth will return the hash of information to the auth/:provider/callback in the Rack environment under the key omniauth.auth so this is what you can use in your desired way like creating a entry to your database and storing the current user in session params.

get '/auth/:provider/callback' do
    erb "
    <h1>Hello #{request.env['omniauth.auth']['info']['name']}</h1>"
end

post '/auth/developer/callback' do
    erb "
    <h1>Hello #{request.env['omniauth.auth']['info']['name']}</h1>"
end

Here, POST request method is used for developer strategy and GET request method for twitter and gitlab as that is how their working is defined in their respective strategies.

Failure route:

If user authentication fails on the provider side, OmniAuth will catch the response and then redirect the request to the path /auth/failure, passing a corresponding error message in a parameter named message.

get '/auth/failure' do
    halt(403, erb("<h2>Authentication Failed</h2><h4>Reason: </h4><pre>#{params[:message]}</pre>"))
end

Final result

That’s it, our application is all set to be tested. And this is how it will be working.

Here, I have already logged into my respective accounts so page for entring credentials does not show up but if the user has not logged into his account, he would be first asked to log in.

For complete code, you can check out: OmniAuth with Sinatra Tutorial

So this completes the tutorial. I hope it helps others who are looking to integrate their Ruby Applications with OmniAuth. If you have any feedback, feel free to let me know.

See you next time!

To start writing about updates regarding my GSoC project, the first obvious thing I need to do is to explain what my project really is. So let’s get started.

About my project

What is debci?

Directly stating from the official docs:

The Debian continuous integration (debci) is an automated system that coordinates the execution of automated tests against packages in the Debian system.

Let’s try decoding it:

Debian is a huge system with thousands of packages and within these packages exist inter-package dependencies. So if any package is updated, it is important to test if that package is working correctly but it is equally important to test that all the packages which are dependent on this updated package are working correctly too.

Debci is a platform serving this purpose of automated testing for the entire Debian archive whenever a new version of the package, or of any package in its dependency chain is available. It comes with a UI that lets developers easily run tests and see their results if they pass or not.

For my GSoC project, I am working to implement some incremental improvements to debci making it easier to use and maintain.

Community Bonding Period

The debci community

Everyone I have come across till now in the community is very nice. The debci community in itself is a small but active community. It really feels good to be a part of conversations here.

Weekly call set up

I have two mentors for this project Antonio Terceiro and Paul Gevers and they have set up a weekly sync call with me in which I will share my updates regarding the work done in the past week, any issues I am facing, and discuss the work for next week. In addition to this, I can always contact them on IRC for any issue I am stuck in.

Work till now

The first thing I did in the community bonding period is setting up this blog. I wanted to have one for a long time and this seems to be a really nice opportunity to start. And the fact this has been added to Planet Debian too makes me happier to write. I am still trying to get a hang of this and definitely need to learn how to spend less time writing it.

I also worked on my already opened merge requests during this period and got them merged.

Since I am already familiar with the codebase, so I started with my first deliverable a bit earlier before the official coding period begins which is migrating the logins to Salsa, Debian’s Gitlab Instance.

Currently, debci uses Debian SSO client certificates for logging in, but that is deprecated so it needs to be migrated to use Salsa as an identity provider.

The OmniAuth library is being used to implement this with help of ruby-omniauth-gitlab strategy. I explored a great deal about integrating OmniAuth with our application and bumped into so many issues too when I began implementing that. Once I am done integrating the Salsa Authentication with debci, I am planning to write a separate tutorial on that which could be helpful to other people using OmniAuth with their application.

With that, the community bonding period has ended on 7th June and the coding period officially begins and for now, I will be continuing working on my first deliverable.

That’s all for now. See you next time!

I am really excited that my Google Summer of Code proposal with Debian for the project “Debian Continuous Integration improvements” has been accepted. Through this blog, I am here to share about my Pre-GSoC journey.

I knew about GSoC since my first year of college but had this misconception that only great coders get selected for GSoC which did not let me apply to the program until my 3rd year of engineering. I applied this year not because I thought I have turned into one but because I actually wanted to give a fair try to this before the time I become ineligible to participate.

Finding Project

Scrolling through the list of GSoC 2021 organizations, I was checking out projects of organizations I am familiar with. Debian is one of the huge Open Source communities that has always inspired developers around the world to contribute to Open Source. So as I checked through Debian projects, I got excited to find the “Debian Continuous Integration improvements” project (referred to as debci in this post) related to web development and more concerned with backend work which is something I am very much interested in.

I joined the community, and as directed by the application tasks of the project I set up the debci on my machine and started with an issue labeled as a newcomer. Soon I was able to submit my first Merge Request and it was reviewed by Antonio Terceiro, the mentor of my GSoC Project. With his further guidance, I was able to turn MR into an acceptable patch, and voila it got merged! That really did boost my morale to contribute further to the project.

Student Application Period

At the suggestion of the mentor, during the Student Application Period, I worked on more open issues which were helping me understand the codebase better and in turn the deliverables of the project for my proposal. For my every doubt, I first tried to tackle it myself, and if still not able to find a solution I turn to mentors who did their best to answer my queries and this is how I completed my proposal and got it reviewed by Antonio before finally submitting it on 13th April. I still cannot express that feeling of satisfaction I achieved on submitting the proposal. I finally successfully applied for GSoC.

After Proposal Submission

I did not stop my contribution after the Student Application period ended and kept on working on more issues which helped me stay in touch with the project and also because I was enjoying it a lot. I had already made up my mind to contribute more in Open Source as I learned and enjoyed plenty during this process.

Day of Results

On 17th May, I got my acceptance mail at around 11:30 pm at night and I remember screaming and waking everybody up in my home to announce the news to them. It was truly the happiest moment for me.

Moment of Truth

I would admit that I got involved with GSoC because of the reputation associated with it but things I learned during this pre GSoC period have made me realize the fun and learning opportunities associated with working opensource and I am here to stay for sure.

I plan to write more blogs regarding my project and keep you guys updated about my work. Stay tuned!