1 of 5

Setting Grafana Dashboard with Parquet

Before proceeding, please create the function in the region where you have the S3 Bucket.

Click on 'Create Function'.

Once in the 'Create Function' window:

Select 'Author from scratch.
In Basic information:
- Define function name.
- Select Runtime as '.NET 6 (C#/PowerShell) (from the drop-down options).
- Select Architecture, x86_64 (from the given options).
In Permissions, set the default execution role for Lambda functions:
- If you have already created a role, you can continue by selecting the existing role.
  - Select 'Use an existing role'.
  -  Select the Existing role as 'influx_lambda_role' from the drop-down options.

Select 'Create function'.

Once the function is created, click on 'Upload from'.

Click 'Upload'.

Select the function code's zip file, and click 'Open'.

Once the zip file is uploaded, click 'Save'.

Once on the function window, under the 'Code' tab, locate 'Runtime settings' and click 'Edit' if you wish to edit any credentials.

In Runtime settings:

Add Runtime as .NET 6 (C#/PowerShell).
Change handler as AWSLambdaFileConvert::AWSLambdaConvert.Function::ConvertFiles
Chose Architecture.
Click 'Save'.

To edit the general configuration, click' Configuration' and select' Edit'.

Edit details such as memory, ephemeral (temporary) storage, and time out.

Click 'Save'.

Adding S3 Trigger for Lambda

Click on 'Add trigger'

Note:

These parameters may change depending on the DBC use, such as the size of the DBC.

Select the following

Source as S3.
Select the Bucket to which the logger uploads data.
Suffix add (.rxd).
Acknowledge the warning.

The trigger is successfully added.

Configuring conversion settings

Upload the FileConvert.json file into the root folder of the Bucket.

Enable the required flags.

Editable example:

{
  "InfluxDB": {
    "enabled": false,
    "url": "",
    "token": "",
    "org": "",
    "bucket": ""
  },
  "CSV": {
    "enabled": false,
    "resampling": {
      "enabled": true,
      "rate": 1000
    }
  },
  "AmazonTimestream": {
    "enabled": false,
    "db_name": "test_demo",
    "table_name": "rxddata",
    "downsampling": true,
    "downsampling_tolerance": 0.01
  },
  "Snapshot": {
    "enabled": false,
    "db_name": "dbname",
    "table_name": "snapshot"
  },
  "MDF": {
    "enabled": false,
    "usecompression": false
  },
  "BLF": {
    "enabled": false
  },
  "Parquet": {
    "enabled": true,
    "resampling": {
      "enabled": true,
      "rate": 1000
    }
  },
  "Synch": {
    "enabled": false,
    "main_logger": "",
    "addon_loger1": "",
    "lastfolder": ""
  },
  "RxdSettings": {
    "MaxTimeGap": 0
  }
}

Configuring AWS Glue settings

Setting a Grafana Dashboard using the Athena plugin

Configuring AWS Glue settings

Open AWS Glue services.

Create a database.

Name the database and click 'Create database'.

A database with the desired name is created.

Open IAM services and select 'Role'.

Select the desired roles.

Select the trusted entity, choose the use case as 'glue', and select 'Next'.

In Permissions, set the default execution role for the Glue function, and click 'Next'.

Name, review and create:

Name the role.
Add description (if needed/as required).
Click 'Create role'.

Click 'ETL jobs'.

Create a script by clicking the 'Script editor' tab.

Select 'Spark' from the options under Engine (from the drop-down).

Enable 'Upload script' and click 'Choose file'.

View or edit your script code (if required)

Add the basic properties under the 'Job details' tab.

Once the properties are added, click 'Create schedule'.

Add Schedule properties and click 'Create schedule.'

Click the 'Run' tab and then click 'Run job' to manually run the job; the database will be populated if the parquet files are already available in S3.

Note:

The database will be populated only if the parquet files are already available in S3; otherwise, there is no need to run the job manually.

The run screen then displays the active status of the job.

The job is successfully scheduled.

The database details, along with the logger number, are displayed.

Note:

Happens only when the job is run manually after creation it will have to wait until the first run.

Click on the logger serial number to get a detailed report.

Setting a Grafana Dashboard using the Athena plugin

Visit , click 'data sources' and then click 'Add new data source'.

Get plugin >> Amazon Athena >> Install Plugin

'Amazon Athena' plugin is added,

Return to AWS IAM settings, select the 'Users tab', and click 'Create User. '

Specify the user details and click 'Next'.

To set permissions, select 'Attach policies directly'.

Select from the existing policies.

Click 'Create user'

Create the Access Key.

Select 'Others' and click 'Next.'

Set Description tags if required, and click 'Create access key'.

Once the Access key is created, copy and save both the access and the secret access key, or download them as a .csv file and click 'Done'.

Now go back to Grafana to specify the Athena dashboard's settings and click 'Save and test.' Ensure that the output location is a folder in the S3 bucket.

Once the data source is tested, this message is displayed.

Click 'Dashboards', click 'New' and then click 'New dashboard'.

Click 'Add visualization'.

Select the data source.

This opens the basic dashboard screen.

Add your query and then click 'Run query'.

For example

Editable Format

The Dashboard is created.

Setting Grafana Dashboard with Parquet

Before proceeding, please create the function in the region where you have the S3 Bucket.

Click on 'Create Function'.

Once in the 'Create Function' window:

Select 'Author from scratch.
In Basic information:
- Define function name.
- Select Runtime as '.NET 6 (C#/PowerShell) (from the drop-down options).
- Select Architecture, x86_64 (from the given options).
In Permissions, set the default execution role for Lambda functions:
- If you are uploading for the first time, make sure you have created an IAM role with the following permissions; refer to the
- If you have already created a role, you can continue by selecting the existing role.
  - Select 'Use an existing role'.
  -  Select the Existing role as 'influx_lambda_role' from the drop-down options.

Select 'Create function'.

Once the function is created, click on 'Upload from'.

Click 'Upload'.

Select the function code's zip file, and click 'Open'.

https://itltdgithub.s3.ap-south-1.amazonaws.com/Lambda_Function/Lambda_Function_V10.zip

Once the zip file is uploaded, click 'Save'.

Once on the function window, under the 'Code' tab, locate 'Runtime settings' and click 'Edit' if you wish to edit any credentials.

In Runtime settings:

Add Runtime as .NET 6 (C#/PowerShell).
Change handler as AWSLambdaFileConvert::AWSLambdaConvert.Function::ConvertFiles
Chose Architecture.
Click 'Save'.

To edit the general configuration, click' Configuration' and select' Edit'.

Edit details such as memory, ephemeral (temporary) storage, and time out.

Click 'Save'.

Configuring AWS Glue settings

Open AWS Glue services.

Create a database.

Name the database and click 'Create database'.

A database with the desired name is created.

Open IAM services and select 'Role'.

Select the desired roles.

Select the trusted entity, choose the use case as 'glue', and select 'Next'.

In Permissions, set the default execution role for the Glue function, and click 'Next'.

Name, review and create:

Name the role.
Add description (if needed/as required).
Click 'Create role'.

Click 'ETL jobs'.

Create a script by clicking the 'Script editor' tab.

Select 'Spark' from the options under Engine (from the drop-down).

Enable 'Upload script' and click 'Choose file'.

Locate the on your device, select and open.

View or edit your script code (if required)

Add the basic properties under the 'Job details' tab.

Once the properties are added, click 'Create schedule'.

Add Schedule properties and click 'Create schedule.'

Click on the 'CRON expressions' to learn more about CRON syntax for assigning a required schedule.

Click the 'Run' tab and then click 'Run job' to manually run the job; the database will be populated if the parquet files are already available in S3.

Note:

The database will be populated only if the parquet files are already available in S3; otherwise, there is no need to run the job manually.

The run screen then displays the active status of the job.

The job is successfully scheduled.

The database details, along with the logger number, are displayed.

Note:

Happens only when the job is run manually after creation it will have to wait until the first run.

Click on the logger serial number to get a detailed report.

Setting a Grafana Dashboard using the Athena plugin

Visit , click 'data sources' and then click 'Add new data source'.

Get plugin >> Amazon Athena >> Install Plugin

'Amazon Athena' plugin is added,

Return to AWS IAM settings, select the 'Users tab', and click 'Create User. '

Specify the user details and click 'Next'.

To set permissions, select 'Attach policies directly'.

Select from the existing policies.

Click 'Create user'

Create the Access Key.

Select 'Others' and click 'Next.'

Set Description tags if required, and click 'Create access key'.

Once the Access key is created, copy and save both the access and the secret access key, or download them as a .csv file and click 'Done'.

Now go back to Grafana to specify the Athena dashboard's settings and click 'Save and test.' Ensure that the output location is a folder in the S3 bucket.

Once the data source is tested, this message is displayed.

Click 'Dashboards', click 'New' and then click 'New dashboard'.

Click 'Add visualization'.

Select the data source.

This opens the basic dashboard screen.

Add your query and then click 'Run query'.

For example

Editable Format

SELECT
    $__timeGroup(time, $__interval) as Time,
    AVG("$__column") as Alt
FROM
    "$__table"
WHERE
    $__timeFilter(time)
GROUP BY
    $__timeGroup(time, $__interval)
ORDER BY
    Time ASC