Data Import

Data Import

Data Import Checklist

Before importing data, you should be able to answer the following questions about the current format of your file.

  1. What type of file are you importing?
  2. Is the data normalised or pivoted?
  3. Do the data variables you wish to import already exist in the DataSight database? If not, you can choose to add them before you begin, or you can choose to add them in Step 2 of the Import process.
  4. To assist with mapping your data, check that your file contains a Title Row for all the data fields you wish to import. If not, you may wish to add header rows with title names for all your columns of data prior to import.
  5. Are there standards entered against which your data can be compared?
  6. Before starting the import process, make sure you have closed the import file.
  7. It is advisable to only import blocks of data that have the same collection information. If this collection information changes within a data file, you may need to undertake separate imports for subsets of the data to vary the collection information assigned to each record.

 

The data import routine takes you through five steps to prepare your data for import into DataSight.

  • Step 1: Define and Format Data
  • Step 2: Filter and Map your Data
  • Step 3: Preview Data
  • Step 4: Finalise your import
  • Step 5: Commit your Data

 

Pivoted data layout for data importation from a file

This format will allow you to import the entire spreadsheet including stations and data into DataSight. A pivoted table allows you to extract the significance from a large, detailed data set. In DataSight Version 1 or 2, this was the most efficient file format for manual data importation.

Geographic location, mapped as type 'Level'

Date & time information, mapped as type  'Date Time'

Measured parameters with data, mapped as type 'Variable Data'

Additional columns containing metadata, mapped as type 'Metadata'

Area

Monitoring Site

Date

Time

Var 1

Var 2

Var 3

Var 4

Var 5

Var 6

Var 7

Var 8

Var 9

Var 10 etc

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

In the above spreadsheet, Area is mapped to Level 2 and could be a geographic area or a type of monitoring that is undertaken. The Monitoring Site is the actual physical monitoring station or Level 3 where data is captured. You also need the date when the data was captured and the actual time on the specific date that the data was captured.

Var 1 to Var 4 columns are mapped to each DataSight variable and the values in each cell imported to each respective DataSight variable. Var 5 to Var 10 columns are mapped to each DataSight metadata variable and the values in each cell imported to their respective DataSight metadata variable for each measured parameter Var 1 to Var 4.

See Map Variables in a Pivoted Dataset to map pivoted imports.

 

Normalised data layout for data importation from a file

DataSight version 3 can be used to manually and automatically import normalised data. Normalised data occurs when each row in a spreadsheet contains only one parameter value. For environmental data we might expect to see columns for locality, datetime, a parameter name and its measured value. For water quality data we might also expect additional columns containing metadata relating to the measured parameter, such as the minimum detection level.

Project

Area

Monitoring Site

Date

Time

Parameter

Value

Metadata 1

Metadata 2

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

In the above spreadsheet, Project is mapped to Level 1, Area is mapped to Level 2, the Monitoring Site is Level 3, date and time are mapped accordingly. Parameters are mapped using a Variable Mapping template, with metadata columns mapped to a metadata variable in the same way as mapping variables and metadata variables is performed in a pivoted data layout.

See Map Variables in a Normalised Dataset to map normalised imports.

 

Data migration issues

In DataSight, each entry or record MUST have a unique location and datetime. When capturing environmental data you may collect many records for the same date at one specific location. For example, if you are monitoring changes in water temperature at one site, but at various depths, the location and date will remain constant. This is fine, but each record must have a unique timestamp. In this instance it is imperative that the time changes for each of the depth entries. This can be as simple as varying each entry (depth) by one second, such as 19:50:00, 19:50:01, 19:50:02, and so on...

There are two options when you are faced with data such as this.Data

Option 1. Ensure time values are assigned correctly before importing. This may involve adjusting equipment settings to record separate timestamps for each sample, or editing the data manually before importing in DataSight.

Option 2. DataSight can assign time values during import (see Map Levels).

 

Refer to Data Migration for further issues concerning data importation and database structure within DataSight.

    • Related Articles

    • Video: Import Normalised Data

      This video provides a basic overview of normalised data import.

    • Automatically Import Data

      Using saved Import Templates, an import routine can be automated as a task. From within the Tasks grid view, either double-click on a selected task row, or click on the Edit The Edit Tasks dialogue window appears. Select an Import Template (See also ...

    • Commit Your Import

      In the last step of the Import Routine, you will be able to view a summary of your import specifications prior to committing your data or document to the database. At the top of the Import Summary section, DataSight will display a coloured text box ...

    • Import Data

      Open a desired Station Table. Click Import Data. Select Manual Data Entry or Import Data File. Manual Data Entry Enter date, time and your variable value. Click Save. Import Data File Browse to your file and select Open, then click Upload. Configure ...

    • Preview Data

      In Step 3 of the Data Import Routine, preview the data you are about to import. Click Back to step backwards through the Import Routine to change any formatting, filtering or mapping as required. Click Next if you are satisfied with the data set. See ...