Method and system for uniformly managing AI models based on distributed file system

Abstract

The invention discloses a method and system for unified management of AI models based on a distributed file system. Based on the distributed file system, a model iteration management module is additionally arranged to extract preset model file information; wherein the model file information comprises information such as a model name, a model version, model creation time, whether a model is onlinepublic and whether the model is dirty, an AI model record is newly added in the metadata table, and the model is stored in a model warehouse according to a preset model storage path to construct an AImodel management system formed by combining the metadata table and the model warehouse. The newly-added model reading module analyzes data input by a user, extracts model information, matches the model information with records in a metadata table, extracts metadata items in the table, checks whether a model is online or not, and if the model is online, extracts complete target model information including a dirty model state from nodes of the distributed file system according to metadata and returns the complete target model information to the user. A user can use the model in real time and can also optimize the model and upload the model again to facilitate optimization and updating of the model.

Description

technical field [0001] The invention relates to the technical field of artificial intelligence, in particular to a method and system for unified management of AI models based on a distributed file system. Background technique [0002] Currently, there are various methods applied to generate predictive models. Data scientists and engineers can choose from a variety of languages ​​to build AI predictive models. For example, use the Python language to call the scikit-learn framework to build a prediction model, use Java or Scala language to call the Spark MLlib framework to build a prediction model, and so on. Numerous construction methods result in specialized models for various environments. Recently, with the widespread use of deep learning, frameworks such as TensorFlow and Pytorch support publishing machine learning models through REST or gRPC, but traditional machine learning frameworks, such as scikit-learn, gensim, xgboost, etc., do not yet support it. Also, tensorfl...

AI Technical Summary

Problems solved by technology

However, this method has many shortcomings. For example, the process of converting TensorFlow’s proprietary model to PMML format model is cumbersome, and the model file after conversion generally becomes larger, and requires the installation of corresponding plug-ins to read, which increases the difficulty of deployment; and PMML The unified model does not record the unique optimization of each framework, and the running speed is slow

In addition, there may be deviations between the transformed model and the predicted value of the original model.

Images