Re: [PATCH 1/3] aws: add dynamic cloud configuration support using AWS CLI

On Thu, Sep 04, 2025 at 09:55:46AM -0400, Chuck Lever wrote:
> On 9/4/25 5:00 AM, Luis Chamberlain wrote:
> > Add support for dynamically generating AWS instance types and regions
> > configuration using the AWS CLI, similar to the Lambda Labs implementation.
> > 
> > This allows users to:
> > - Query real-time AWS instance availability
> > - Generate Kconfig files with current instance families and regions
> > - Choose between dynamic and static configuration modes
> > - See pricing estimates and resource summaries
> > 
> > Key components:
> > - scripts/aws-cli: AWS CLI wrapper tool for kdevops
> > - scripts/aws_api.py: Low-level AWS API functions
> > - Updated generate_cloud_configs.py to support AWS
> > - Makefile integration for AWS Kconfig generation
> > - Option to use dynamic or static AWS configuration
> > 
> > Usage: Run 'make cloud-config' to generate dynamic configuration.
> 
> I'd like to see more documentation for this make target. Is this a
> target to be run as part of every workflow, or is it one that developers
> run every once in a while? (I think the latter, based on the next patch
> in this series, but it would be nice to put that in docs somewhere).
> 
> For example, ISTR a docs file that describes "make refs-default".

Since it takes ~ 6 minutes to generate AWS cloud configs, this likely
means other cloud providers will likely take the same amount of time.
If we parallelize we still end up with 6 minutes.

So to not waste those cycles we can also leveage the make refs-default
practice so we keep in-tree fresh refs, but allow the user to kick off
more of a fresh listing. In the case of cloud configs, we shouldn't
expect users wanting to do this full generation all the time. So we can
save users time by putting the onus on:

make cloud-config
make cloud-update 

Onto kdevops cloud maintainers.

> > This also parallelize cloud provider operations to significantly improve
> > generation.
> > 
> > $ time make cloud-config
> > Cloud Provider Configuration Summary
> > ============================================================
> > 
> > ✓ Lambda Labs: 14/20 instances available, 14 regions, $0.50-$10.32/hr
> >   Kconfig files generated successfully
> > 
> > ✓ AWS: 979 instance types available, 17 regions, ~$0.05-$3.60/hr
> >   Kconfig files generated successfully
> > 
> > ⚠ Azure: Dynamic configuration not yet implemented
> > ⚠ GCE: Dynamic configuration not yet implemented
> > 
> > Note: Dynamic configurations query real-time availability
> > Run 'make menuconfig' to configure your cloud provider
> > 
> > real    6m51.859s
> > user    37m16.347s
> > sys     3m8.130s
> 
> I spent a little time yesterday adding new instance families by hand,
> after asking the AWS instance type assistant to recommend appropriate
> families for CI/CD. I added: t3, t3a, m7i-flex, and the two g4 families.
> Other families looked too expensive or might be more than is needed for
> CI/CD (like who needs 16 GPUs, 192 vCPUs, and 8 200GbE adapters for a
> development system? ;-)
> 
> What I'd like to do is prevent an overload of choices in the instance
> menus -- perhaps we can maintain a list somewhere of the families we'd
> like to add to the menu and let the scripting consult that list as it
> constructs the menu.

The burden wasn't so much on user interface. So I'd prefer the full
listing, why not.

> (And, I can drop my by-hand patches... I didn't realize you were ready
> to post your automation that does the same thing).

I didn't know I was doing this until last night I needed a GPU on AWS.
Instead of writing new instnaces by hand I figured I'd give this a shot.
Turns out that telling Cluade to learn from the Lambda patches helped a
lot. And so did the docs for it and its dynamic cloud support. So all
that paid off.

I suspect it should therefore be just as easy for azure too. I'm not
touching azure, as I don't have a need yet.

> > This also adds support for GPU AMIs:
> 
> IMHO the new GPU AMI support is a good idea, but should be split into a
> separate patch in this series.

Sure.

> >  .gitignore                                    |    3 +
> >  defconfigs/aws-gpu-g6e-ai                     |   53 +
> >  .../templates/aws/terraform.tfvars.j2         |    5 +
> >  scripts/aws-cli                               |  436 +++++++
> >  scripts/aws_api.py                            | 1135 +++++++++++++++++
> 
> Note there is also an Ansible collection that provides the AWS API
> to playbooks: amazon.aws. This is not a recommendation to
> reimplement this patch, just pointing out there are other ways to
> skin the cat.

Sure. Went with skinning option A. We can evaluate alternatives on a
need basis to scale.

  Luis