Is there anything like this but for analyzing thousands of models? Most of this type of tool seem to work best for iterating and experimenting, but I'd actually like to use it for tracking and debugging something in production.

We have a service that runs thousands of custom, user-defined models and when something goes awry having this level of insight would be pretty useful.

Glad to see that the code[1] is AGPL-3.0 licensed.

[1]: https://github.com/transformerlab/transformerlab-app

Could this be run as a web interface? For example, getting a GCP instance with GPU and deploying it there and accessing it from the browser.

EDIT: Seems it has a client/server model and the server can be installed somewhere else: https://transformerlab.ai/docs/install/advanced-install

But I'm wondering if this is an electron based app and I could run the client on a web interface as well.

Mmm interesting, curious to check it out. The feature list starts with ‘Pick a model’, which I think is the hardest part in the first place. How do you know what mode to pick?

Nit: the website is cut off left and right on mobile (ios)

Seems quite comprehensive. I wonder if there’s a way to “escape” from the system. It looks nice for getting started but in practice at some point you’ll want programmatic access for e.g. advanced API features.

E.g. a well behaving system independent pipeline serialization could work. Didn’t find anything about this with a quick glance of the docs.

Edit: with a bit longer glance there is a plugin system that allows adding custom code quite easily.