Re: KCIDB database

> It is not that straight forward to update python version to python3.

Of course, and you cannot really do that, as KernelCI Legacy needs Python 2.
I was rather suggesting you install Python 3 alongside Python 2, if possible
for your target.

> This upgrade will be disruptive for our testing and deployment unless the
> KCIDB results push can be done with python2.7 which we will prefer at this
> point of time.

If this is hard, one (radical?) way to do this could be running Python 3 on a
different machine (in a container?) and executing kcidb-submit there, via e.g.
ssh (docker run?).

But perhaps others have other ideas.

Nick

On 10/18/24 5:04 PM, Manoj Kumar wrote:
> We used ansible playbooks to deploy backend server https://github.com/
> kernelci/kernelci-backend-config <https://github.com/kernelci/kernelci-
> backend-config>
> 
> Which create virtual environment based on python2.7
> 
> https://github.com/kernelci/kernelci-backend-config/
> commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071 <https://github.com/kernelci/
> kernelci-backend-config/commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071>
> 
>  
> 
> It is not that straight forward to update python version to python3.
> 
> This upgrade will be disruptive for our testing and deployment unless the
> KCIDB results push can be done with python2.7 which we will prefer at this
> point of time.
> 
>  
> 
> br,
> 
> Manoj
> 
>  
> 
> *From: *Nikolai Kondrashov <[email protected]>
> *Date: *Friday, 18 October 2024 at 14:32
> *To: *Manoj Kumar <[email protected]>
> *Cc: *[email protected] <[email protected]>, Denys
> Fedoryshchenko <[email protected]>
> *Subject: *Re: KCIDB database
> 
> Hi Manoj,
> 
>> I looked at the below link to get installation steps for the kcidb-submit
> command.
>>
>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/
> kcidb/installation/> <https://docs.kernelci.org/
>> kcidb/installation/>
>>
>> This instruction is for Python version V3.9 but our backend is using the
>> Python 2.7.16 at the moment.
>>
>> Do you have instruction for python 2.7.16 to install this package
> 
> KCIDB requires Python 3, and that's why KernelCI Legacy (which requires Python
> 2) is invoking the kcidb-submit command-line tool in a new process instead of
> using the KCIDB library directly.
> 
> You will need to install Python 3 in your deployment, I assume. Perhaps Denys,
> or someone else on the KernelCI maillist (both CC'd) could tell you how we did
> that.
> 
> Nick
> 
> On 10/18/24 4:21 PM, Manoj Kumar wrote:
>> Hi Nikolai,
>> 
>>  
>> 
>> I am planning to push test build data to public KCIDB . I am seeing a failure
>> because of the kcidb-submit command in a thread at backend.
>> 
>>  
>> 
>> The class KcidbSubmit expect the command kcidb-submit installed on the backend
>> system.
>> 
>> /class KcidbSubmit(object):/
>> 
>> /    def __init__(self, kcidb_options):/
>> 
>> /        kcidb_path = kcidb_options.get("kcidb_path", "")/
>> 
>> /        self.kcidb_submit_cmd = os.path.join(kcidb_path, "kcidb-submit")/
>> 
>>  
>> 
>> I looked at the below link to get installation steps for the kcidb-submit command.
>> 
>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/
> kcidb/installation/> <https://docs.kernelci.org/
>> kcidb/installation/>
>> 
>>  
>> 
>> This instruction is for Python version V3.9 but our backend is using the
>> Python 2.7.16 at the moment.
>> 
>> Do you have instruction for python 2.7.16 to install this package
>> 
>>  
>> 
>> Br,
>> 
>> Manoj
>> 
>>  
>> 
>> *From: *Nikolai Kondrashov <[email protected]>
>> *Date: *Tuesday, 8 October 2024 at 11:46
>> *To: *Manoj Kumar <[email protected]>
>> *Subject: *Re: KCIDB database
>> 
>> Hi Manoj,
>> 
>> On 10/8/24 1:26 PM, Manoj Kumar wrote:
>>> One additional question regarding the credentials file.
>>>
>>> Attached credentials files was shared with me named  “.kernelci-production-ci-
>>> arm.json” .
>>>
>>> The attached file contains multiple parameters , If I understood correctly all
>>> the contents of this file should be assigned to this variable https://
>>> github.com/kernelci/kernelci-backend/
>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https://
>>> github.com/kernelci/kernelci-backend/
>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>> 
>> Not the contents, but a path to the file itself.
>> 
>>> Also , the email ID used by internal kernelCI deployment is different . They
>>> will be “[email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected]>>>
>>> and [email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected]>>>”.
>> 
>> That doesn't matter, the email in the JSON credentials is simply a user
>> (service account) ID and is specific to Google Cloud. It identifies a
>> particular submitter to the system.
>> 
>> You don't really need to look into the credentials file at all.
>> 
>> Nick
>> 
>> On 10/8/24 1:26 PM, Manoj Kumar wrote:
>>> Hi Nick,
>>> 
>>>  
>>> 
>>> Thanks for your prompt responses :thumbsup
>>> 
>>>  
>>> 
>>> One additional question regarding the credentials file.
>>> 
>>> Attached credentials files was shared with me named  “.kernelci-production-ci-
>>> arm.json” .
>>> 
>>> The attached file contains multiple parameters , If I understood correctly all
>>> the contents of this file should be assigned to this variable https://
>>> github.com/kernelci/kernelci-backend/
>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https://
>>> github.com/kernelci/kernelci-backend/
>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>> 
>>>  
>>> 
>>> Also , the email ID used by internal kernelCI deployment is different . They
>>> will be “[email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected]>>>
>>> and [email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected]>>>”.
>>> 
>>>  
>>> 
>>> Br,
>>> 
>>> Manoj
>>> 
>>>  
>>> 
>>>  
>>> 
>>> *From: *Nikolai Kondrashov <[email protected]>
>>> *Date: *Tuesday, 8 October 2024 at 05:36
>>> *To: *Manoj Kumar <[email protected]>
>>> *Subject: *Re: KCIDB database
>>> 
>>> On 10/7/24 7:49 PM, Manoj Kumar wrote:
>>>> I do have some questions if you can help me find answer.
>>>>
>>>>   * Do I need to use the private key from the credential json for this
>>>>     parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/
> kernelci-backend/> <https://github.com/kernelci/
>> kernelci-backend/> <https://github.com/kernelci/
>>> kernelci-backend/>
>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38
>>>>     <https://github.com/kernelci/kernelci-backend/
>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>> 
>>> Yes, that needs to be pointing to the credentials JSON file.
>>> 
>>>>   * As the storage server is internal , I guess we need to open access to
>>>>     storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->
> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>
>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>
>>>>     backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/
>>>>     kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/
>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173>
>>>>     files so the kcidb server can access them . Can you confirm this , please
>>> 
>>> Yes, KCIDB right now will try to download every 256th URL you submit, that is
>>> under 5MB. The artifact caching system is in trial mode still. However, what's
>>> more important, *people* will try to open the URLs you submit.
>>> 
>>> Opening the access to the whole server might be problematic for you guys, as
>>> you want to keep some results secret, IIRC. Maybe you could copy them to some
>>> other web server and translate the URLs as you submit them?
>>> 
>>> Nick
>>> 
>>> On 10/7/24 7:49 PM, Manoj Kumar wrote:
>>>> Hi Nick,
>>>> 
>>>> Nice to meet you too mate, I am going through the code base of kcidb.py to
>>>> push results.
>>>> 
>>>> I will test pushing results to playground with the earlier credentials as you
>>>> mentioned.
>>>> 
>>>>  
>>>> 
>>>> Also , I will be configuring internal kernelci deployment to push results to
>>>> public KCIDB for builds and tests.
>>>> 
>>>>  
>>>> 
>>>> I do have some questions if you can help me find answer.
>>>> 
>>>>   * Do I need to use the private key from the credential json for this
>>>>     parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/
> kernelci-backend/> <https://github.com/kernelci/
>> kernelci-backend/> <https://github.com/kernelci/
>>> kernelci-backend/>
>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38
>>>>     <https://github.com/kernelci/kernelci-backend/
>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>>>   * As the storage server is internal , I guess we need to open access to
>>>>     storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->
> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>
>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>
>>>>     backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/
>>>>     kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/
>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173>
>>>>     files so the kcidb server can access them . Can you confirm this , please
>>>> 
>>>>  
>>>> 
>>>> Br,
>>>> 
>>>> Manoj
>>>> 
>>>>  
>>>> 
>>>> *From: *Nikolai Kondrashov <[email protected]>
>>>> *Date: *Monday, 7 October 2024 at 12:15
>>>> *To: *Manoj Kumar <[email protected]>
>>>> *Subject: *Re: KCIDB database
>>>> 
>>>> Hi Manoj,
>>>> 
>>>> Nice to meet you!
>>>> 
>>>> I'm glad to hear there's progress towards resuming KCIDB submissions at ARM!
>>>> 
>>>> The access parameters are out of date, although they will work for
>>>> submissions. Here's more up-to-date info:
>>>> 
>>>> https://docs.kernelci.org/kcidb/submitter_guide/ <https://docs.kernelci.org/
> kcidb/submitter_guide/> <https://docs.kernelci.org/
>> kcidb/submitter_guide/> <https://docs.kernelci.org/
>>> kcidb/submitter_guide/> <https://docs.kernelci.org/
>>>> kcidb/submitter_guide/>
>>>> 
>>>> The credentials I gave ARM before should still work. I'd recommend restarting
>>>> submissions to the "playground" database ("playground_kcidb_new" topic) so you
>>>> can experiment freely without worrying about bothering anyone with invalid
>>>> data. Then when you're ready, switch to production ("kcidb_new").
>>>> 
>>>> Don't hesitate to reach out, if you have any questions. I'll add you as the
>>>> ARM's current contact here:
>>>> 
>>>> https://github.com/kernelci/kcidb/issues/328 <https://github.com/kernelci/
> kcidb/issues/328> <https://github.com/kernelci/
>> kcidb/issues/328> <https://github.com/kernelci/
>>> kcidb/issues/328> <https://github.com/kernelci/
>>>> kcidb/issues/328>
>>>> 
>>>> I think ARM doesn't let employees use Slack, but if I'm wrong, I can invite
>>>> you to our channel, so we can have more interactive Q&A, if needed.
>>>> 
>>>> Nick
>>>> 
>>>> On 10/7/24 12:37 PM, Manoj Kumar wrote:
>>>>> Hi Nikolai,
>>>>> 
>>>>> I wanted to say hello to you and start the initial thread regarding pushing
>>>>> the test results to KernelCI KCIDB.
>>>>> I will be enabling the changes to push kernel test and build results to KCIDB
>>>>> from ARM testing . I was told by Cristian and Mark that you are the POC from
>>>>> kernelCI KCIDB side.
>>>>> 
>>>>> The access parameters shared with me are below , can you confirm they are
>>>>> still valid and not expired / updated .
>>>>> 
>>>>>  
>>>>> 
>>>>>  
>>>>> 
>>>>> The access parameters are:
>>>>> 
>>>>>          Google Cloud project name: kernelci-production
>>>>>            Playground dataset name: playground_kernelci04
>>>>> Playground submission queue topic: playground_kernelci_new
>>>>>            Production dataset name: kernelci04
>>>>> Production submission queue topic: kernelci_new
>>>>> 
>>>>>  
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Manoj
>>>>> 
>>>>>  
>>>>> 
>>>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>>>> confidential and may also be privileged. If you are not the intended
>>>>> recipient, please notify the sender immediately and do not disclose the
>>>>> contents to any other person, use it for any purpose, or store or copy the
>>>>> information in any medium. Thank you.
>>>> 
>>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>>> confidential and may also be privileged. If you are not the intended
>>>> recipient, please notify the sender immediately and do not disclose the
>>>> contents to any other person, use it for any purpose, or store or copy the
>>>> information in any medium. Thank you.
>>> 
>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>> confidential and may also be privileged. If you are not the intended
>>> recipient, please notify the sender immediately and do not disclose the
>>> contents to any other person, use it for any purpose, or store or copy the
>>> information in any medium. Thank you.
>> 
>> IMPORTANT NOTICE: The contents of this email and any attachments are
>> confidential and may also be privileged. If you are not the intended
>> recipient, please notify the sender immediately and do not disclose the
>> contents to any other person, use it for any purpose, or store or copy the
>> information in any medium. Thank you.
> 
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.