Committing your own modules and using them in a workflow

Now that we’ve created a new workflow with existing modules, deployed and ran it, let’s see how you can add your own code to a Ryax workflow.

All along this example, we’ll use the “Twitter analysis” workflow you’ve built in the previous section of this guide, and swap the Text Classifier module with a new custom module you (figuratively) developed yourself:

../_images/46263c58-96ef-4c56-89c4-c786f4e328a0.png

We will use a model we’ll call “Named Entity Recognition (BERT)”, a state of the art BERT language model to classify the content of Tweets across several categories:

  • O, Outside of a named entity

  • B-MIS, Beginning of a miscellaneous entity right after another miscellaneous entity

  • I-MIS, Miscellaneous entity

  • B-PER, Beginning of a person’s name right after another person’s name

  • I-PER, Person’s name

  • B-ORG, Beginning of an organisation right after another organisation

  • I-ORG, Organisation

  • B-LOC, Beginning of a location right after another location

  • I-LOC, Location

More information about this model can be found here.

Let’s start by importing this new module into Ryax.

Note

The following section focuses on one specific workflow example (“Twitter analysis”). For more universal guidelines on how to create custom workflows with your own code in Ryax, please refer to the “Ryax Tour” and “How-To” sections of this documentation.

Committing external code to Ryax

Scanning Git repositories

Ryax enables you to create your own modules from code that was developed outside the platform. For this purpose, we’ll use the “Repositories” screen.

One of the most standard ways Data Scientists (and developers in general) store, version, review and collaborate on their code is using Git repositories. Leveraging this common practice, Ryax provides an interface able to pull code from any Git environment to create custom modules that can be used in Ryax workflows.

Let’s navigate the “Repositories” section of the platform using the left menu bar:

../_images/087cb2f6-fe79-47ab-8417-d85ded15c663.png

This screen allows you to save and manage a list of repositories you wish to pull your code from:

../_images/ab1751a8-0ecc-4806-bcb0-b02edf7408b6.png

To list a new repository, click the “New repository” button at the top right. You’ll be asked to enter a repo name and its URL:

../_images/ec6dba9a-a469-4550-97b0-78d173043f0e.png

For this guide, you can use the following public Gitlab directory we’re providing: https://gitlab.com/ryax-tech/workflows/nlp.git

Warning

Your repository URL must end with the “.git” extension for Ryax to properly interpret the repo’s path.

After entering you repo’s name and URL, you can click the “Scan repository” button to start searching for code to pull:

../_images/09569aa1-d63d-418d-b600-6083a98df773.png

You’ll then be asked to enter your credentials to allow access for Ryax to crawl your repo. For this example, we’ll use the following login and password:

  • Username: anonymous

  • Password: anonymous

../_images/3fecab65-99b2-4fce-a364-a1208404fb2e.png

Then, you can hit the “Scan repository” button to initiate the repository scan:

../_images/ba46ec3b-c71d-4ed4-a7d6-c475646d5880.png

Ryax will crawl the Git URL you provided and list all recognizable modules in the “Scan results” panel below:

../_images/5d584938-8aae-4f4f-9e9d-f8cfd9afef91.png

Note

For more information on how to build recognizable modules for Ryax, please refer to the “How-to” section of this documentation.

Reviewing the result list, we can see the detected module: “Named Entity Recognition (BERT for token classification). Let’s say this is the one we’ve just developed and wants to add to our workflow.

Now you may launch this specific module’s build by clicking on the lightning button to the right:

../_images/32354009-5320-48b3-afd8-f6f7af0826ef.png

This “Build module” command will launch the actual code pull into Ryax. The platform will automatically pull your module’s code, package it in a productizable way and publish it in the Ryax store for future use in any workflow.

Building modules

After scanning repositories, you’ll be able to monitor all scanned modules in the “Module builds” section of the Ryax platform:

../_images/3400260d-c067-4dad-9bb7-251f0e3278ca.png

You’ll then be able to view a list of all scanned modules, manage it, and launch module builds:

../_images/63cae44a-4fda-4464-99ac-2d1c73071c90.png

You can use the contextual buttons on the right of the list to:

../_images/8f19e0f4-7eb0-458e-aceb-9ca94f587f22.png

Build/or retry build a module.

../_images/7d0dc8c5-5050-4839-b4af-0ff7cb6ddc00.png

Access a module's documentation (if built).

../_images/41fba96c-2bbe-48d4-aafa-af15e1ea0505.png

Delete a module scan.

In-list statuses inform you about module build states at any moment:

../_images/2b32a88a-5d0c-4e77-96b9-fe4fc1b83af9.png

After a few seconds, your “Named Entity Recognition (BERT for token classification)” module should display the “Success” status:

../_images/7c2d2126-5ce5-4895-95f0-e1e84c84a559.png

When successfully built, your module will appear in the Ryax module store as well as the contextual module panel of the Studio editor. You’ll then be able to drop it into a workflow.

Working with the Ryax store

Searching the store

You can find the module you’ve just built using the search bar at the top:

../_images/33338962-5a2a-4fb4-aefc-71a114971b19.png

Each listed module displays quick view information to help you navigate through existing sources, processors and publishers:

../_images/3325498d-817d-40b6-91c8-f5b2d8dbd8d9.png

Accessing modules’ resources

By clicking on a module, you can access detailed information and documentation:

../_images/b30d29ec-2e85-4fc5-81e6-bc500c869651.png

This screen displays the modules’ full descriptions, as well as inputs and outputs details.

Using your modules in a custom workflow

Let’s move back to the Studio Editor and add our new module to the workflow. Open the Module Store pane to the right and search for “Recognition…” to filter out the module list:

../_images/f7c2f002-2403-4d2c-8f9e-85f982d940be.png

You can then drag this new module in your workflow:

../_images/2d79b197-4a93-4fda-a1fd-7141a3c56627.png

Since we will use this new module instead of the ‘old’ classifier, let’s remove the old one from our workflow:

../_images/9d9b205e-76bf-4d6c-b5e4-d96cddb86804.png

This will delete the module as well as its upstream & downstream links, so you can now interleave the new module in place and interlink it:

../_images/b0921205-8c8b-477d-b5c5-ff8424db6a91.png

Last step is to configure it by specifying the previous module’s output (i.e. a ‘reference value’ in the Ryax vocable) as the current module’s input:

../_images/a85168f2-7b91-4c80-a583-dc9a24a6d74a.png   ../_images/8653dd1f-41ac-4494-a165-95c26723290b.png

[Optional: if you have set up the “Publish data to Google Sheet” module, you’ll have to re-configure its input as well since the relative upstream module changed. Proceed as follows.]

../_images/64df61dd-bff9-4718-a06e-9707b7fcbce7.png

This is all you had to do to replace the ‘old’ module by the new one. You’re now ready to deploy your workflow and have a look at its executions:

../_images/9e794527-c6d6-4393-a501-6c6f5be3dc46.png

If you navigate to the execution dashboard and you select one of the “Named Entity Recognition” module’s executions, you’ll be able to view the workflow’s results in the log window (see red rectangle below):

../_images/439f68c0-601b-4fa1-af33-56d243301262.png

In this example, we can see that the algorithm identified the keyword “Edge AI” in the Tweet, with a confidence index of “96.96564674377441%”, and so on:

../_images/c68d7050-22b0-4446-9aad-c150208aab23.png

[Optional] If you set up the Google Sheet publisher module, you will be able to access your workflow’s results straight from your Google Sheet, which provides a more advanced way to visualize data:

../_images/128f9763-aea1-4acb-8cce-a69200168526.png