Re: KCIDB database

On 10/24/24 6:13 PM, Manoj Kumar wrote:
> Any suggestion for the right approach with your past experiences with KCIDB

I really never had to do that, but after checking with your distro's docs, and
finding that having both Python 2 and Python 3 installed together is
impossible (if that's indeed true), my next step would be to try with Docker,
using the official Python 3 container images as the base.

Nick

On 10/24/24 6:13 PM, Manoj Kumar wrote:
> Any suggestion for the right approach with your past experiences with KCIDB
> 
>  
> 
> Br,
> 
> Manoj
> 
>  
> 
> *From: *Nikolai Kondrashov <[email protected]>
> *Date: *Friday, 18 October 2024 at 16:05
> *To: *Manoj Kumar <[email protected]>
> *Cc: *[email protected] <[email protected]>, Denys
> Fedoryshchenko <[email protected]>
> *Subject: *Re: KCIDB database
> 
>> It is not that straight forward to update python version to python3.
> 
> Of course, and you cannot really do that, as KernelCI Legacy needs Python 2.
> I was rather suggesting you install Python 3 alongside Python 2, if possible
> for your target.
> 
>> This upgrade will be disruptive for our testing and deployment unless the
>> KCIDB results push can be done with python2.7 which we will prefer at this
>> point of time.
> 
> If this is hard, one (radical?) way to do this could be running Python 3 on a
> different machine (in a container?) and executing kcidb-submit there, via e.g.
> ssh (docker run?).
> 
> But perhaps others have other ideas.
> 
> Nick
> 
> On 10/18/24 5:04 PM, Manoj Kumar wrote:
>> We used ansible playbooks to deploy backend server https://github.com/ <https://github.com/>
>> kernelci/kernelci-backend-config <https://github.com/kernelci/kernelci-
>> backend-config>
>> 
>> Which create virtual environment based on python2.7
>> 
>> https://github.com/kernelci/kernelci-backend-config/ <https://github.com/
> kernelci/kernelci-backend-config/>
>> commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071 <https://github.com/kernelci/
>> kernelci-backend-config/commit/52d442759cc29ef2453fa5c9a2c90d3fc2c74071>
>> 
>>  
>> 
>> It is not that straight forward to update python version to python3.
>> 
>> This upgrade will be disruptive for our testing and deployment unless the
>> KCIDB results push can be done with python2.7 which we will prefer at this
>> point of time.
>> 
>>  
>> 
>> br,
>> 
>> Manoj
>> 
>>  
>> 
>> *From: *Nikolai Kondrashov <[email protected]>
>> *Date: *Friday, 18 October 2024 at 14:32
>> *To: *Manoj Kumar <[email protected]>
>> *Cc: *[email protected] <[email protected]>, Denys
>> Fedoryshchenko <[email protected]>
>> *Subject: *Re: KCIDB database
>> 
>> Hi Manoj,
>> 
>>> I looked at the below link to get installation steps for the kcidb-submit
>> command.
>>>
>>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/
> kcidb/installation/> <https://docs.kernelci.org/
>> kcidb/installation/> <https://docs.kernelci.org/
>>> kcidb/installation/>
>>>
>>> This instruction is for Python version V3.9 but our backend is using the
>>> Python 2.7.16 at the moment.
>>>
>>> Do you have instruction for python 2.7.16 to install this package
>> 
>> KCIDB requires Python 3, and that's why KernelCI Legacy (which requires Python
>> 2) is invoking the kcidb-submit command-line tool in a new process instead of
>> using the KCIDB library directly.
>> 
>> You will need to install Python 3 in your deployment, I assume. Perhaps Denys,
>> or someone else on the KernelCI maillist (both CC'd) could tell you how we did
>> that.
>> 
>> Nick
>> 
>> On 10/18/24 4:21 PM, Manoj Kumar wrote:
>>> Hi Nikolai,
>>> 
>>>  
>>> 
>>> I am planning to push test build data to public KCIDB . I am seeing a failure
>>> because of the kcidb-submit command in a thread at backend.
>>> 
>>>  
>>> 
>>> The class KcidbSubmit expect the command kcidb-submit installed on the backend
>>> system.
>>> 
>>> /class KcidbSubmit(object):/
>>> 
>>> /    def __init__(self, kcidb_options):/
>>> 
>>> /        kcidb_path = kcidb_options.get("kcidb_path", "")/
>>> 
>>> /        self.kcidb_submit_cmd = os.path.join(kcidb_path, "kcidb-submit")/
>>> 
>>>  
>>> 
>>> I looked at the below link to get installation steps for the kcidb-submit command.
>>> 
>>> https://docs.kernelci.org/kcidb/installation/ <https://docs.kernelci.org/
> kcidb/installation/> <https://docs.kernelci.org/
>> kcidb/installation/> <https://docs.kernelci.org/
>>> kcidb/installation/>
>>> 
>>>  
>>> 
>>> This instruction is for Python version V3.9 but our backend is using the
>>> Python 2.7.16 at the moment.
>>> 
>>> Do you have instruction for python 2.7.16 to install this package
>>> 
>>>  
>>> 
>>> Br,
>>> 
>>> Manoj
>>> 
>>>  
>>> 
>>> *From: *Nikolai Kondrashov <[email protected]>
>>> *Date: *Tuesday, 8 October 2024 at 11:46
>>> *To: *Manoj Kumar <[email protected]>
>>> *Subject: *Re: KCIDB database
>>> 
>>> Hi Manoj,
>>> 
>>> On 10/8/24 1:26 PM, Manoj Kumar wrote:
>>>> One additional question regarding the credentials file.
>>>>
>>>> Attached credentials files was shared with me named  “.kernelci-production-ci-
>>>> arm.json” .
>>>>
>>>> The attached file contains multiple parameters , If I understood correctly all
>>>> the contents of this file should be assigned to this variable https://
>>>> github.com/kernelci/kernelci-backend/
>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https://
>>>> github.com/kernelci/kernelci-backend/
>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>> 
>>> Not the contents, but a path to the file itself.
>>> 
>>>> Also , the email ID used by internal kernelCI deployment is different . They
>>>> will be “[email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected]>>>>
>>>> and [email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected]>>>>”.
>>> 
>>> That doesn't matter, the email in the JSON credentials is simply a user
>>> (service account) ID and is specific to Google Cloud. It identifies a
>>> particular submitter to the system.
>>> 
>>> You don't really need to look into the credentials file at all.
>>> 
>>> Nick
>>> 
>>> On 10/8/24 1:26 PM, Manoj Kumar wrote:
>>>> Hi Nick,
>>>> 
>>>>  
>>>> 
>>>> Thanks for your prompt responses :thumbsup
>>>> 
>>>>  
>>>> 
>>>> One additional question regarding the credentials file.
>>>> 
>>>> Attached credentials files was shared with me named  “.kernelci-production-ci-
>>>> arm.json” .
>>>> 
>>>> The attached file contains multiple parameters , If I understood correctly all
>>>> the contents of this file should be assigned to this variable https://
>>>> github.com/kernelci/kernelci-backend/
>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38 <https://
>>>> github.com/kernelci/kernelci-backend/
>>>> blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>>> 
>>>>  
>>>> 
>>>> Also , the email ID used by internal kernelCI deployment is different . They
>>>> will be “[email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected]>>>>
>>>> and [email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected] <mailto:[email protected]>>>>”.
>>>> 
>>>>  
>>>> 
>>>> Br,
>>>> 
>>>> Manoj
>>>> 
>>>>  
>>>> 
>>>>  
>>>> 
>>>> *From: *Nikolai Kondrashov <[email protected]>
>>>> *Date: *Tuesday, 8 October 2024 at 05:36
>>>> *To: *Manoj Kumar <[email protected]>
>>>> *Subject: *Re: KCIDB database
>>>> 
>>>> On 10/7/24 7:49 PM, Manoj Kumar wrote:
>>>>> I do have some questions if you can help me find answer.
>>>>>
>>>>>   * Do I need to use the private key from the credential json for this
>>>>>     parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/
> kernelci-backend/> <https://github.com/kernelci/
>> kernelci-backend/> <https://github.com/kernelci/
>>> kernelci-backend/> <https://github.com/kernelci/
>>>> kernelci-backend/>
>>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38
>>>>>     <https://github.com/kernelci/kernelci-backend/
>>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>>> 
>>>> Yes, that needs to be pointing to the credentials JSON file.
>>>> 
>>>>>   * As the storage server is internal , I guess we need to open access to
>>>>>     storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->
> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>
>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>
>>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>>
>>>>>     backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/
>>>>>     kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/
>>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173>
>>>>>     files so the kcidb server can access them . Can you confirm this , please
>>>> 
>>>> Yes, KCIDB right now will try to download every 256th URL you submit, that is
>>>> under 5MB. The artifact caching system is in trial mode still. However, what's
>>>> more important, *people* will try to open the URLs you submit.
>>>> 
>>>> Opening the access to the whole server might be problematic for you guys, as
>>>> you want to keep some results secret, IIRC. Maybe you could copy them to some
>>>> other web server and translate the URLs as you submit them?
>>>> 
>>>> Nick
>>>> 
>>>> On 10/7/24 7:49 PM, Manoj Kumar wrote:
>>>>> Hi Nick,
>>>>> 
>>>>> Nice to meet you too mate, I am going through the code base of kcidb.py to
>>>>> push results.
>>>>> 
>>>>> I will test pushing results to playground with the earlier credentials as you
>>>>> mentioned.
>>>>> 
>>>>>  
>>>>> 
>>>>> Also , I will be configuring internal kernelci deployment to push results to
>>>>> public KCIDB for builds and tests.
>>>>> 
>>>>>  
>>>>> 
>>>>> I do have some questions if you can help me find answer.
>>>>> 
>>>>>   * Do I need to use the private key from the credential json for this
>>>>>     parameter https://github.com/kernelci/kernelci-backend/ <https://github.com/kernelci/
> kernelci-backend/> <https://github.com/kernelci/
>> kernelci-backend/> <https://github.com/kernelci/
>>> kernelci-backend/> <https://github.com/kernelci/
>>>> kernelci-backend/>
>>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38
>>>>>     <https://github.com/kernelci/kernelci-backend/
>>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L38>
>>>>>   * As the storage server is internal , I guess we need to open access to
>>>>>     storage server for these urls https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->
> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>
>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>
>>> <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci- <https://github.com/kernelci/kernelci->>>>
>>>>>     backend/blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/
>>>>>     kcidb.py#L173 <https://github.com/kernelci/kernelci-backend/
>>>>>     blob/7e7bff2703c73b2a3a3734b7bcd0593cbe657705/app/utils/kcidb.py#L173>
>>>>>     files so the kcidb server can access them . Can you confirm this , please
>>>>> 
>>>>>  
>>>>> 
>>>>> Br,
>>>>> 
>>>>> Manoj
>>>>> 
>>>>>  
>>>>> 
>>>>> *From: *Nikolai Kondrashov <[email protected]>
>>>>> *Date: *Monday, 7 October 2024 at 12:15
>>>>> *To: *Manoj Kumar <[email protected]>
>>>>> *Subject: *Re: KCIDB database
>>>>> 
>>>>> Hi Manoj,
>>>>> 
>>>>> Nice to meet you!
>>>>> 
>>>>> I'm glad to hear there's progress towards resuming KCIDB submissions at ARM!
>>>>> 
>>>>> The access parameters are out of date, although they will work for
>>>>> submissions. Here's more up-to-date info:
>>>>> 
>>>>> https://docs.kernelci.org/kcidb/submitter_guide/ <https://docs.kernelci.org/
> kcidb/submitter_guide/> <https://docs.kernelci.org/
>> kcidb/submitter_guide/> <https://docs.kernelci.org/
>>> kcidb/submitter_guide/> <https://docs.kernelci.org/
>>>> kcidb/submitter_guide/> <https://docs.kernelci.org/
>>>>> kcidb/submitter_guide/>
>>>>> 
>>>>> The credentials I gave ARM before should still work. I'd recommend restarting
>>>>> submissions to the "playground" database ("playground_kcidb_new" topic) so you
>>>>> can experiment freely without worrying about bothering anyone with invalid
>>>>> data. Then when you're ready, switch to production ("kcidb_new").
>>>>> 
>>>>> Don't hesitate to reach out, if you have any questions. I'll add you as the
>>>>> ARM's current contact here:
>>>>> 
>>>>> https://github.com/kernelci/kcidb/issues/328 <https://github.com/kernelci/
> kcidb/issues/328> <https://github.com/kernelci/
>> kcidb/issues/328> <https://github.com/kernelci/
>>> kcidb/issues/328> <https://github.com/kernelci/
>>>> kcidb/issues/328> <https://github.com/kernelci/
>>>>> kcidb/issues/328>
>>>>> 
>>>>> I think ARM doesn't let employees use Slack, but if I'm wrong, I can invite
>>>>> you to our channel, so we can have more interactive Q&A, if needed.
>>>>> 
>>>>> Nick
>>>>> 
>>>>> On 10/7/24 12:37 PM, Manoj Kumar wrote:
>>>>>> Hi Nikolai,
>>>>>> 
>>>>>> I wanted to say hello to you and start the initial thread regarding pushing
>>>>>> the test results to KernelCI KCIDB.
>>>>>> I will be enabling the changes to push kernel test and build results to KCIDB
>>>>>> from ARM testing . I was told by Cristian and Mark that you are the POC from
>>>>>> kernelCI KCIDB side.
>>>>>> 
>>>>>> The access parameters shared with me are below , can you confirm they are
>>>>>> still valid and not expired / updated .
>>>>>> 
>>>>>>  
>>>>>> 
>>>>>>  
>>>>>> 
>>>>>> The access parameters are:
>>>>>> 
>>>>>>          Google Cloud project name: kernelci-production
>>>>>>            Playground dataset name: playground_kernelci04
>>>>>> Playground submission queue topic: playground_kernelci_new
>>>>>>            Production dataset name: kernelci04
>>>>>> Production submission queue topic: kernelci_new
>>>>>> 
>>>>>>  
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Manoj
>>>>>> 
>>>>>>  
>>>>>> 
>>>>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>>>>> confidential and may also be privileged. If you are not the intended
>>>>>> recipient, please notify the sender immediately and do not disclose the
>>>>>> contents to any other person, use it for any purpose, or store or copy the
>>>>>> information in any medium. Thank you.
>>>>> 
>>>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>>>> confidential and may also be privileged. If you are not the intended
>>>>> recipient, please notify the sender immediately and do not disclose the
>>>>> contents to any other person, use it for any purpose, or store or copy the
>>>>> information in any medium. Thank you.
>>>> 
>>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>>> confidential and may also be privileged. If you are not the intended
>>>> recipient, please notify the sender immediately and do not disclose the
>>>> contents to any other person, use it for any purpose, or store or copy the
>>>> information in any medium. Thank you.
>>> 
>>> IMPORTANT NOTICE: The contents of this email and any attachments are
>>> confidential and may also be privileged. If you are not the intended
>>> recipient, please notify the sender immediately and do not disclose the
>>> contents to any other person, use it for any purpose, or store or copy the
>>> information in any medium. Thank you.
>> 
>> IMPORTANT NOTICE: The contents of this email and any attachments are
>> confidential and may also be privileged. If you are not the intended
>> recipient, please notify the sender immediately and do not disclose the
>> contents to any other person, use it for any purpose, or store or copy the
>> information in any medium. Thank you.
> 
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.