Telemac DroneMDT (Metadata) Notebook
This notebook aims to understand how to: - generate flight report from drone data (i.e: acquired with DJI Mavic 2 Pro) - generate Metadata to publish to Zenodo and SDI platform (flight reports and csv's entities are separated to allow on site generation and deeper analysis when back to office) - publish to Zenodo to obtain DOI on RAW data - process in WebODM (OpenDroneMap) - download results from OpenDroneMap - publish a new DOI version on Zenodo with generated output from photogrammetryCommon variables:
[1]:
# Script folder
dronemdt_folder = '/home/sylvain/DroneMDT'
# Session / Flight data input
input = '/home/sylvain/HDD5To/DATA/TELEMAC'
# Output geoflow config (json and csv entities)
output = '/home/sylvain/HDD5To/DATA/TELEMAC'
# Geoflow folder where .env is stored and jobs will be available
geoflow_folder = '/home/sylvain/geoflow-g2oi'
# Geoserver configuration
WS = 'OSUR'
DS = 'telemac'
# Zenodo setup
sandbox = False
# Move to dronemdt_folder
import os
os.chdir(dronemdt_folder)
Part 1a: Check images validity
[5]:
# DATA have to be copied from Drone internal storage or sdcard DCIM folder using suh naming convention:
# YYYYMMDD_COUNTRYCODE-optionalplace_device_session-number
# ├── DCIM : folder to store videos and photos depending on the media collected
#
# Check validity of images copied from Drone to disk
# rclone copy cloudIRD:/G2OI/Missions/Aldabra /media/HDD5To/DATA/ALDABRA/
[2]:
from check_images import *
img_list = im_list(input)
for i in range(len(img_list)):
check_image_with_pil(img_list[i])
Part 1b: Generate report
[4]:
## Using subprocess:
import subprocess
subprocess.check_output(["Rscript", "./generate_drone_metadata.R", input])
# from rpy2.robjects.packages import importr
Loading required package: magrittr
uasimg (version 1.8.3)
Bug reports: https://github.com/ucanr-igis/uasimg/issues
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_1/DCIM
- Looking for image files (all image files)
- Found DJI Mavic 2 Pro Hasselblad
- Running exiftool (this can take a while)...Done.
- Total file size: 2870 MB
- Creating footprints...Done.
- Computing forward overlap...Done.
- Cache saved
- Metadata file not found
All done
2023-05-24_07-40-23_321JPEGs
- computing thumbnail file names...Done.
- thumbnails already exist
- Done.
processing file: uas_report_metrics.Rmd
output file: uas_report_metrics.knit.md
Output created: METADATA/Report_2023-05-24_07-40-23_321JPEGs.html
Done.
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2/DCIM
- Looking for image files (all image files)
- Found DJI Mavic 2 Pro Hasselblad
- Running exiftool (this can take a while)...Done.
- Total file size: 3475 MB
- Creating footprints...Done.
- Computing forward overlap...Done.
- Cache saved
- Metadata file not found
All done
2023-05-24_09-08-51_339JPEGs
- computing thumbnail file names...Done.
- thumbnails already exist
- Done.
processing file: uas_report_metrics.Rmd
output file: uas_report_metrics.knit.md
Output created: METADATA/Report_2023-05-24_09-08-51_339JPEGs.html
Done.
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_3/DCIM
- Looking for image files (all image files)
- Found DJI Mavic 2 Pro Hasselblad
- Running exiftool (this can take a while)...Done.
- Total file size: 3243 MB
- Creating footprints...Done.
- Computing forward overlap...Done.
- Cache saved
- Metadata file not found
All done
2023-05-24_09-33-18_335JPEGs
- computing thumbnail file names...Done.
- thumbnails already exist
- Done.
processing file: uas_report_metrics.Rmd
output file: uas_report_metrics.knit.md
Output created: METADATA/Report_2023-05-24_09-33-18_335JPEGs.html
Done.
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4/DCIM
- Looking for image files (all image files)
- Found DJI Mavic 2 Pro Hasselblad
- Running exiftool (this can take a while)...Done.
- Total file size: 1788 MB
- Creating footprints...Done.
- Computing forward overlap...Done.
- Cache saved
- Metadata file not found
All done
2023-05-24_09-55-02_192JPEGs
- computing thumbnail file names...Done.
- thumbnails already exist
- Done.
processing file: uas_report_metrics.Rmd
output file: uas_report_metrics.knit.md
Output created: METADATA/Report_2023-05-24_09-55-02_192JPEGs.html
Done.
/home/sylvain/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1/DCIM
- Looking for image files (all image files)
- Found DJI Mavic 2 Pro Hasselblad
- Running exiftool (this can take a while)...Done.
- Total file size: 3455 MB
- Creating footprints...Done.
- Computing forward overlap...Done.
- Cache saved
- Metadata file not found
All done
2023-07-05_10-33-06_276JPEGs
- computing thumbnail file names...Done.
- thumbnails already exist
- Done.
processing file: uas_report_metrics.Rmd
output file: uas_report_metrics.knit.md
Output created: METADATA/Report_2023-07-05_10-33-06_276JPEGs.html
Done.
/home/sylvain/HDD5To/DATA/TELEMAC/20230802_REU-ermitage_UAV-02_1/DCIM
- Looking for image files (all image files)
- Found DJI Mavic 2 Pro Hasselblad
- Running exiftool (this can take a while)...Done.
- Total file size: 3213 MB
- Creating footprints...Done.
- Computing forward overlap...Done.
- Cache saved
- Metadata file not found
All done
2023-08-02_06-59-36_351JPEGs
- computing thumbnail file names...Done.
- thumbnails already exist
- Done.
processing file: uas_report_metrics.Rmd
output file: uas_report_metrics.knit.md
Output created: METADATA/Report_2023-08-02_06-59-36_351JPEGs.html
Done.
[4]:
b'[1] "/home/sylvain/HDD5To/DATA/TELEMAC"\n[1] "processing: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_1"\n[1] "METADATA folder available"\n1/27 \n2/27 [unnamed-chunk-1] \n3/27 \n4/27 [unnamed-chunk-2] \n5/27 \n6/27 [setup] \n7/27 \n8/27 [data_uris_insert]\n9/27 \n10/27 [show_flight_info]\n11/27 \n12/27 [leaflet_ctr] \n13/27 \n14/27 [fp_tab] \n15/27 \n16/27 [leaflet_fp] \n17/27 \n18/27 [kml_links] \n19/27 \n20/27 [nothingtodo] \n21/27 \n22/27 [unnamed-chunk-3] \n23/27 \n24/27 [unnamed-chunk-4] \n25/27 \n26/27 [unnamed-chunk-5] \n27/27 \n/usr/bin/pandoc +RTS -K512m -RTS uas_report_metrics.knit.md --to html4 --from markdown+autolink_bare_uris+tex_math_single_backslash --output /media/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_1/METADATA/Report_2023-05-24_07-40-23_321JPEGs.html --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/pagebreak.lua --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/latex-div.lua --self-contained --variable bs3=TRUE --section-divs --template /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable theme=bootstrap --mathjax --variable \'mathjax-url=https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML\' --include-in-header /tmp/RtmpW91q3Q/rmarkdown-strd15a224ccd0e.html \n[1] "processing: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2"\n[1] "METADATA folder available"\n1/27 \n2/27 [unnamed-chunk-1] \n3/27 \n4/27 [unnamed-chunk-2] \n5/27 \n6/27 [setup] \n7/27 \n8/27 [data_uris_insert]\n9/27 \n10/27 [show_flight_info]\n11/27 \n12/27 [leaflet_ctr] \n13/27 \n14/27 [fp_tab] \n15/27 \n16/27 [leaflet_fp] \n17/27 \n18/27 [kml_links] \n19/27 \n20/27 [nothingtodo] \n21/27 \n22/27 [unnamed-chunk-3] \n23/27 \n24/27 [unnamed-chunk-4] \n25/27 \n26/27 [unnamed-chunk-5] \n27/27 \n/usr/bin/pandoc +RTS -K512m -RTS uas_report_metrics.knit.md --to html4 --from markdown+autolink_bare_uris+tex_math_single_backslash --output /media/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2/METADATA/Report_2023-05-24_09-08-51_339JPEGs.html --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/pagebreak.lua --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/latex-div.lua --self-contained --variable bs3=TRUE --section-divs --template /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable theme=bootstrap --mathjax --variable \'mathjax-url=https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML\' --include-in-header /tmp/RtmpW91q3Q/rmarkdown-strd15a39cd0e78.html \n[1] "processing: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_3"\n[1] "METADATA folder available"\n1/27 \n2/27 [unnamed-chunk-1] \n3/27 \n4/27 [unnamed-chunk-2] \n5/27 \n6/27 [setup] \n7/27 \n8/27 [data_uris_insert]\n9/27 \n10/27 [show_flight_info]\n11/27 \n12/27 [leaflet_ctr] \n13/27 \n14/27 [fp_tab] \n15/27 \n16/27 [leaflet_fp] \n17/27 \n18/27 [kml_links] \n19/27 \n20/27 [nothingtodo] \n21/27 \n22/27 [unnamed-chunk-3] \n23/27 \n24/27 [unnamed-chunk-4] \n25/27 \n26/27 [unnamed-chunk-5] \n27/27 \n/usr/bin/pandoc +RTS -K512m -RTS uas_report_metrics.knit.md --to html4 --from markdown+autolink_bare_uris+tex_math_single_backslash --output /media/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_3/METADATA/Report_2023-05-24_09-33-18_335JPEGs.html --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/pagebreak.lua --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/latex-div.lua --self-contained --variable bs3=TRUE --section-divs --template /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable theme=bootstrap --mathjax --variable \'mathjax-url=https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML\' --include-in-header /tmp/RtmpW91q3Q/rmarkdown-strd15a66a31802.html \n[1] "processing: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4"\n[1] "METADATA folder available"\n1/27 \n2/27 [unnamed-chunk-1] \n3/27 \n4/27 [unnamed-chunk-2] \n5/27 \n6/27 [setup] \n7/27 \n8/27 [data_uris_insert]\n9/27 \n10/27 [show_flight_info]\n11/27 \n12/27 [leaflet_ctr] \n13/27 \n14/27 [fp_tab] \n15/27 \n16/27 [leaflet_fp] \n17/27 \n18/27 [kml_links] \n19/27 \n20/27 [nothingtodo] \n21/27 \n22/27 [unnamed-chunk-3] \n23/27 \n24/27 [unnamed-chunk-4] \n25/27 \n26/27 [unnamed-chunk-5] \n27/27 \n/usr/bin/pandoc +RTS -K512m -RTS uas_report_metrics.knit.md --to html4 --from markdown+autolink_bare_uris+tex_math_single_backslash --output /media/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4/METADATA/Report_2023-05-24_09-55-02_192JPEGs.html --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/pagebreak.lua --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/latex-div.lua --self-contained --variable bs3=TRUE --section-divs --template /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable theme=bootstrap --mathjax --variable \'mathjax-url=https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML\' --include-in-header /tmp/RtmpW91q3Q/rmarkdown-strd15a4fb616db.html \n[1] "processing: /home/sylvain/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1"\n[1] "METADATA folder available"\n1/27 \n2/27 [unnamed-chunk-1] \n3/27 \n4/27 [unnamed-chunk-2] \n5/27 \n6/27 [setup] \n7/27 \n8/27 [data_uris_insert]\n9/27 \n10/27 [show_flight_info]\n11/27 \n12/27 [leaflet_ctr] \n13/27 \n14/27 [fp_tab] \n15/27 \n16/27 [leaflet_fp] \n17/27 \n18/27 [kml_links] \n19/27 \n20/27 [nothingtodo] \n21/27 \n22/27 [unnamed-chunk-3] \n23/27 \n24/27 [unnamed-chunk-4] \n25/27 \n26/27 [unnamed-chunk-5] \n27/27 \n/usr/bin/pandoc +RTS -K512m -RTS uas_report_metrics.knit.md --to html4 --from markdown+autolink_bare_uris+tex_math_single_backslash --output /media/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1/METADATA/Report_2023-07-05_10-33-06_276JPEGs.html --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/pagebreak.lua --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/latex-div.lua --self-contained --variable bs3=TRUE --section-divs --template /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable theme=bootstrap --mathjax --variable \'mathjax-url=https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML\' --include-in-header /tmp/RtmpW91q3Q/rmarkdown-strd15a39748203.html \n[1] "processing: /home/sylvain/HDD5To/DATA/TELEMAC/20230802_REU-ermitage_UAV-02_1"\n[1] "METADATA folder available"\n1/27 \n2/27 [unnamed-chunk-1] \n3/27 \n4/27 [unnamed-chunk-2] \n5/27 \n6/27 [setup] \n7/27 \n8/27 [data_uris_insert]\n9/27 \n10/27 [show_flight_info]\n11/27 \n12/27 [leaflet_ctr] \n13/27 \n14/27 [fp_tab] \n15/27 \n16/27 [leaflet_fp] \n17/27 \n18/27 [kml_links] \n19/27 \n20/27 [nothingtodo] \n21/27 \n22/27 [unnamed-chunk-3] \n23/27 \n24/27 [unnamed-chunk-4] \n25/27 \n26/27 [unnamed-chunk-5] \n27/27 \n/usr/bin/pandoc +RTS -K512m -RTS uas_report_metrics.knit.md --to html4 --from markdown+autolink_bare_uris+tex_math_single_backslash --output /media/HDD5To/DATA/TELEMAC/20230802_REU-ermitage_UAV-02_1/METADATA/Report_2023-08-02_06-59-36_351JPEGs.html --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/pagebreak.lua --lua-filter /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/latex-div.lua --self-contained --variable bs3=TRUE --section-divs --template /home/sylvain/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable theme=bootstrap --mathjax --variable \'mathjax-url=https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML\' --include-in-header /tmp/RtmpW91q3Q/rmarkdown-strd15a2c12c0a9.html \n'
Part 2: Generate metadata
[62]:
# Import packages and modules
import os
import shutil
import pandas as pd
import geopandas as gpd
import datetime
# from IPython.display import display
import argparse
import json
import pycountry
os.chdir(dronemdt_folder)
from meteo_helper import meteo
from read_metadata import *
from exif_helper import *
from enriched_func import *
from zenodo_helper import *
from zip_helper import *
from geoflow_helper import *
from raster_helper import *
from process_webodm import *
from settings import *
[11]:
### Filter foder name
folder_name = filter_folder_name(input, 'UAV')
folder_name
[11]:
['20230524_REU-ermitage_UAV-02_4',
'20230524_REU-ermitage_UAV-02_2',
'20230705_REU-ermitage_UAV-02_1',
'20230524_REU-ermitage_UAV-02_3',
'20230802_REU-ermitage_UAV-02_1',
'20230524_REU-ermitage_UAV-02_1']
[12]:
### Put data in basic panda dataframe
df = {}
df = basic_geoflow_df(folder_name)
df.head()
[12]:
| Identifier | Title | Description | Subject | Creator | Date | Type | Language | SpatialCoverage | TemporalCoverage | Relation | Rights | Provenance | Format | Data | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 20230524_REU-ermitage_UAV-02_4 | title:20230524 reu ermitage uav 02 4 | abstract:'No \n\ndescription\n!' | project:None | creator:None | publication:2023-10-22_\nedition:2023-10-22 | dataset | eng | REU-ermitage | None | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | None | resource:image/jpg | None | |
| 1 | 20230524_REU-ermitage_UAV-02_2 | title:20230524 reu ermitage uav 02 2 | abstract:'No \n\ndescription\n!' | project:None | creator:None | publication:2023-10-22_\nedition:2023-10-22 | dataset | eng | REU-ermitage | None | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | None | resource:image/jpg | None | |
| 2 | 20230705_REU-ermitage_UAV-02_1 | title:20230705 reu ermitage uav 02 1 | abstract:'No \n\ndescription\n!' | project:None | creator:None | publication:2023-10-22_\nedition:2023-10-22 | dataset | eng | REU-ermitage | None | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | None | resource:image/jpg | None | |
| 3 | 20230524_REU-ermitage_UAV-02_3 | title:20230524 reu ermitage uav 02 3 | abstract:'No \n\ndescription\n!' | project:None | creator:None | publication:2023-10-22_\nedition:2023-10-22 | dataset | eng | REU-ermitage | None | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | None | resource:image/jpg | None | |
| 4 | 20230802_REU-ermitage_UAV-02_1 | title:20230802 reu ermitage uav 02 1 | abstract:'No \n\ndescription\n!' | project:None | creator:None | publication:2023-10-22_\nedition:2023-10-22 | dataset | eng | REU-ermitage | None | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | None | resource:image/jpg | None |
[13]:
### Define Metadata file
md_file = 'metadata.txt'
### Parse global metadata
mdt_df_glob = global_mdt(input, md_file)
#### Enrich metadata
enrich_drone_mdt(input, df, md_file, mdt_df_glob)
INFO: Global `metadata.txt` found !
WARN: No metadata.txt in /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4/METADATA to parse
INFO: GPS Folder found
Format id to obtain date, country name, optionnal place, and platform
####################
### 2. Enrich Title
####################
INFO: Using glocal title
Images drone, {platform}, {place}, Saint-Gilles, {country} - {date}
format field
####################
### 8. Enrich Language
####################
INFO: Using glocal language
####################
### 4. Enrich Subject
####################
INFO: Using glocal theme
INFO: Using glocal project
####################
### 5. Enrich Creator
####################
INFO: Using glocal author
INFO: Using glocal publisher
INFO: Using glocal owner
INFO: Using glocal pointOfContact
INFO: Using glocal processor
####################
### 11. Enrich Relation
####################
INFO: Using glocal thumbnail
INFO: Using glocal http
####################
### Reading DCIM folder
####################
Extract image coordinates and thumbnails: it could takes some time !
/home/sylvain/DroneMDT/enriched_func.py:14: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
enriched = df_global.iloc[0][keyword][0]
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4
Generate image with a set of images
####################
### 10. Enrich TemporalCoverage
####################
####################
### 00. Prepare Provenance (13) and Description (3)
####################
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4/DCIM/DJI_0070.JPG
####################
Fetching open-meteo
####################
Location: -21.082716650000002, 55.220769149999995
Begin: 2023-05-24 09-53-20 - End: 2023-05-24 10-03-57
Model: best_match
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1966 0 1966 0 0 6676 0 --:--:-- --:--:-- --:--:-- 6664
Convert Dataframe to UTM
Index(['FileName', 'GPSLatitude', 'GPSLongitude', 'GPSAltitude',
'FlightRollDegree', 'FlightPitchDegree', 'FlightYawDegree',
'GimbalRollDegree', 'GimbalPitchDegree', 'GimbalYawDegree',
'ImageWidth', 'ImageLength', 'BitsPerSample', 'ImageDescription',
'Make', 'Model', 'Orientation', 'SamplesPerPixel', 'XResolution',
'YResolution', 'ResolutionUnit', 'Software', 'DateTime',
'YCbCrPositioning', 'ExifOffset', 'GPSVersionID', 'ThumbnailImageWidth',
'ThumbnailImageLength', 'ThumbnailCompression', 'ThumbnailXResolution',
'ThumbnailYResolution', 'ThumbnailResolutionUnit',
'ThumbnailJPEGInterchangeFormat',
'ThumbnailJPEGInterchangeFormatLength', 'ExposureTime', 'FNumber',
'ExposureProgram', 'ISOSpeedRatings', 'ExifVersion', 'DateTimeOriginal',
'DateTimeDigitized', 'ComponentsConfiguration', 'ExposureBiasValue',
'MaxApertureValue', 'MeteringMode', 'LightSource', 'Flash',
'FocalLength', 'FlashPixVersion', 'ColorSpace', 'ExifImageWidth',
'ExifImageLength', 'FileSource', 'SceneType', 'ExposureMode',
'WhiteBalance', 'DigitalZoomRatio', 'FocalLengthIn35mmFilm',
'SceneCaptureType', 'GainControl', 'Contrast', 'Saturation',
'Sharpness', 'DeviceSettingDescription', 'BodySerialNumber',
'ThumbnailImage'],
dtype='object')
Compute convex hull
Export BBOX to GPKG
Exporting geolocation files including thumbnails: it could take some time !
Exporting metadata.csv !
####################
### 13. Enrich Provenance
####################
####################
### 3. Enrich Description
####################
INFO: Using glocal abstract
INFO: Using glocal purpose
INFO: Using glocal info
INFO: Using glocal edition
| └─ 20230524_REU-ermitage_UAV-02_4
|-------- └─ DCIM
|-------- └─ GPS
|---------------- └─ base_2023_05_24_pascal
|---------------- └─ reach_2023_05_24_drone
|------------------------ └─ reachsylvai_raw_202305240249_RINEX_3_03
|------------------------ └─ reachsylvai_raw_202305240330_RINEX_3_03
|---------------- └─ reach_2023_05_24_rover
|-------- └─ METADATA
|---------------- └─ tb
####################
### 14. Enrich Formats
####################
####################
### 15. Enrich data
####################
####################
#### END ####
####################
WARN: No metadata.txt in /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2/METADATA to parse
INFO: GPS Folder found
Format id to obtain date, country name, optionnal place, and platform
####################
### 2. Enrich Title
####################
INFO: Using glocal title
Images drone, {platform}, {place}, Saint-Gilles, {country} - {date}
format field
####################
### 8. Enrich Language
####################
INFO: Using glocal language
####################
### 4. Enrich Subject
####################
INFO: Using glocal theme
INFO: Using glocal project
####################
### 5. Enrich Creator
####################
INFO: Using glocal author
INFO: Using glocal publisher
INFO: Using glocal owner
INFO: Using glocal pointOfContact
INFO: Using glocal processor
####################
### 11. Enrich Relation
####################
INFO: Using glocal thumbnail
INFO: Using glocal http
####################
### Reading DCIM folder
####################
Extract image coordinates and thumbnails: it could takes some time !
/home/sylvain/DroneMDT/enriched_func.py:14: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
enriched = df_global.iloc[0][keyword][0]
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2
Generate image with a set of images
####################
### 10. Enrich TemporalCoverage
####################
####################
### 00. Prepare Provenance (13) and Description (3)
####################
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2/DCIM/DJI_0338.JPG
####################
Fetching open-meteo
####################
Location: -10.5419679, 27.6119689
Begin: 2023-05-24 09-05-25 - End: 2023-05-24 09-26-24
Model: best_match
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1990 0 1990 0 0 7173 0 --:--:-- --:--:-- --:--:-- 7210
Convert Dataframe to UTM
Index(['FileName', 'GPSLatitude', 'GPSLongitude', 'GPSAltitude',
'FlightRollDegree', 'FlightPitchDegree', 'FlightYawDegree',
'GimbalRollDegree', 'GimbalPitchDegree', 'GimbalYawDegree',
'ImageWidth', 'ImageLength', 'BitsPerSample', 'ImageDescription',
'Make', 'Model', 'Orientation', 'SamplesPerPixel', 'XResolution',
'YResolution', 'ResolutionUnit', 'Software', 'DateTime',
'YCbCrPositioning', 'ExifOffset', 'GPSVersionID', 'ThumbnailImageWidth',
'ThumbnailImageLength', 'ThumbnailCompression', 'ThumbnailXResolution',
'ThumbnailYResolution', 'ThumbnailResolutionUnit',
'ThumbnailJPEGInterchangeFormat',
'ThumbnailJPEGInterchangeFormatLength', 'ExposureTime', 'FNumber',
'ExposureProgram', 'ISOSpeedRatings', 'ExifVersion', 'DateTimeOriginal',
'DateTimeDigitized', 'ComponentsConfiguration', 'ExposureBiasValue',
'MaxApertureValue', 'MeteringMode', 'LightSource', 'Flash',
'FocalLength', 'FlashPixVersion', 'ColorSpace', 'ExifImageWidth',
'ExifImageLength', 'FileSource', 'SceneType', 'ExposureMode',
'WhiteBalance', 'DigitalZoomRatio', 'FocalLengthIn35mmFilm',
'SceneCaptureType', 'GainControl', 'Contrast', 'Saturation',
'Sharpness', 'DeviceSettingDescription', 'BodySerialNumber',
'ThumbnailImage'],
dtype='object')
Compute convex hull
Export BBOX to GPKG
Exporting geolocation files including thumbnails: it could take some time !
Exporting metadata.csv !
####################
### 13. Enrich Provenance
####################
####################
### 3. Enrich Description
####################
INFO: Using glocal abstract
INFO: Using glocal purpose
INFO: Using glocal info
INFO: Using glocal edition
| └─ 20230524_REU-ermitage_UAV-02_2
|-------- └─ DCIM
|-------- └─ GPS
|---------------- └─ base_2023_05_24_pascal
|---------------- └─ reach_2023_05_24_drone
|------------------------ └─ reachsylvai_raw_202305240249_RINEX_3_03
|------------------------ └─ reachsylvai_raw_202305240330_RINEX_3_03
|---------------- └─ reach_2023_05_24_rover
|-------- └─ METADATA
|---------------- └─ tb
####################
### 14. Enrich Formats
####################
####################
### 15. Enrich data
####################
####################
#### END ####
####################
WARN: No metadata.txt in /home/sylvain/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1/METADATA to parse
INFO: GPS Folder found
Format id to obtain date, country name, optionnal place, and platform
####################
### 2. Enrich Title
####################
INFO: Using glocal title
Images drone, {platform}, {place}, Saint-Gilles, {country} - {date}
format field
####################
### 8. Enrich Language
####################
INFO: Using glocal language
####################
### 4. Enrich Subject
####################
INFO: Using glocal theme
INFO: Using glocal project
####################
### 5. Enrich Creator
####################
INFO: Using glocal author
INFO: Using glocal publisher
INFO: Using glocal owner
INFO: Using glocal pointOfContact
INFO: Using glocal processor
####################
### 11. Enrich Relation
####################
INFO: Using glocal thumbnail
INFO: Using glocal http
####################
### Reading DCIM folder
####################
Extract image coordinates and thumbnails: it could takes some time !
/home/sylvain/DroneMDT/enriched_func.py:14: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
enriched = df_global.iloc[0][keyword][0]
/home/sylvain/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1
Generate image with a set of images
####################
### 10. Enrich TemporalCoverage
####################
####################
### 00. Prepare Provenance (13) and Description (3)
####################
/home/sylvain/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1/DCIM/DJI_0495.JPG
####################
Fetching open-meteo
####################
Location: -21.082135049999998, 55.221809699999994
Begin: 2023-07-05 10-33-06 - End: 2023-07-05 10-48-40
Model: best_match
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1967 0 1967 0 0 7368 0 --:--:-- --:--:-- --:--:-- 7394
Convert Dataframe to UTM
Index(['FileName', 'GPSLatitude', 'GPSLongitude', 'GPSAltitude',
'FlightRollDegree', 'FlightPitchDegree', 'FlightYawDegree',
'GimbalRollDegree', 'GimbalPitchDegree', 'GimbalYawDegree',
'ImageWidth', 'ImageLength', 'BitsPerSample', 'ImageDescription',
'Make', 'Model', 'Orientation', 'SamplesPerPixel', 'XResolution',
'YResolution', 'ResolutionUnit', 'Software', 'DateTime',
'YCbCrPositioning', 'ExifOffset', 'GPSVersionID', 'ThumbnailImageWidth',
'ThumbnailImageLength', 'ThumbnailCompression', 'ThumbnailXResolution',
'ThumbnailYResolution', 'ThumbnailResolutionUnit',
'ThumbnailJPEGInterchangeFormat',
'ThumbnailJPEGInterchangeFormatLength', 'ExposureTime', 'FNumber',
'ExposureProgram', 'ISOSpeedRatings', 'ExifVersion', 'DateTimeOriginal',
'DateTimeDigitized', 'ComponentsConfiguration', 'ExposureBiasValue',
'MaxApertureValue', 'MeteringMode', 'LightSource', 'Flash',
'FocalLength', 'FlashPixVersion', 'ColorSpace', 'ExifImageWidth',
'ExifImageLength', 'FileSource', 'SceneType', 'ExposureMode',
'WhiteBalance', 'DigitalZoomRatio', 'FocalLengthIn35mmFilm',
'SceneCaptureType', 'GainControl', 'Contrast', 'Saturation',
'Sharpness', 'DeviceSettingDescription', 'BodySerialNumber',
'ThumbnailImage'],
dtype='object')
Compute convex hull
Export BBOX to GPKG
Exporting geolocation files including thumbnails: it could take some time !
Exporting metadata.csv !
####################
### 13. Enrich Provenance
####################
####################
### 3. Enrich Description
####################
INFO: Using glocal abstract
INFO: Using glocal purpose
INFO: Using glocal info
INFO: Using glocal edition
| └─ 20230705_REU-ermitage_UAV-02_1
|-------- └─ DCIM
|-------- └─ GPS
|-------- └─ METADATA
|---------------- └─ tb
####################
### 14. Enrich Formats
####################
####################
### 15. Enrich data
####################
####################
#### END ####
####################
WARN: No metadata.txt in /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_3/METADATA to parse
INFO: GPS Folder found
Format id to obtain date, country name, optionnal place, and platform
####################
### 2. Enrich Title
####################
INFO: Using glocal title
Images drone, {platform}, {place}, Saint-Gilles, {country} - {date}
format field
####################
### 8. Enrich Language
####################
INFO: Using glocal language
####################
### 4. Enrich Subject
####################
INFO: Using glocal theme
INFO: Using glocal project
####################
### 5. Enrich Creator
####################
INFO: Using glocal author
INFO: Using glocal publisher
INFO: Using glocal owner
INFO: Using glocal pointOfContact
INFO: Using glocal processor
####################
### 11. Enrich Relation
####################
INFO: Using glocal thumbnail
INFO: Using glocal http
####################
### Reading DCIM folder
####################
Extract image coordinates and thumbnails: it could takes some time !
/home/sylvain/DroneMDT/enriched_func.py:14: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
enriched = df_global.iloc[0][keyword][0]
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_3
Generate image with a set of images
####################
### 10. Enrich TemporalCoverage
####################
####################
### 00. Prepare Provenance (13) and Description (3)
####################
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_3/DCIM/DJI_0750.JPG
####################
Fetching open-meteo
####################
Location: -21.0821843, 55.221727650000005
Begin: 2023-05-24 09-33-18 - End: 2023-05-24 09-49-53
Model: best_match
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1966 0 1966 0 0 7480 0 --:--:-- --:--:-- --:--:-- 7503
Convert Dataframe to UTM
Index(['FileName', 'GPSLatitude', 'GPSLongitude', 'GPSAltitude',
'FlightRollDegree', 'FlightPitchDegree', 'FlightYawDegree',
'GimbalRollDegree', 'GimbalPitchDegree', 'GimbalYawDegree',
'ImageWidth', 'ImageLength', 'BitsPerSample', 'ImageDescription',
'Make', 'Model', 'Orientation', 'SamplesPerPixel', 'XResolution',
'YResolution', 'ResolutionUnit', 'Software', 'DateTime',
'YCbCrPositioning', 'ExifOffset', 'GPSVersionID', 'ThumbnailImageWidth',
'ThumbnailImageLength', 'ThumbnailCompression', 'ThumbnailXResolution',
'ThumbnailYResolution', 'ThumbnailResolutionUnit',
'ThumbnailJPEGInterchangeFormat',
'ThumbnailJPEGInterchangeFormatLength', 'ExposureTime', 'FNumber',
'ExposureProgram', 'ISOSpeedRatings', 'ExifVersion', 'DateTimeOriginal',
'DateTimeDigitized', 'ComponentsConfiguration', 'ExposureBiasValue',
'MaxApertureValue', 'MeteringMode', 'LightSource', 'Flash',
'FocalLength', 'FlashPixVersion', 'ColorSpace', 'ExifImageWidth',
'ExifImageLength', 'FileSource', 'SceneType', 'ExposureMode',
'WhiteBalance', 'DigitalZoomRatio', 'FocalLengthIn35mmFilm',
'SceneCaptureType', 'GainControl', 'Contrast', 'Saturation',
'Sharpness', 'DeviceSettingDescription', 'BodySerialNumber',
'ThumbnailImage'],
dtype='object')
Compute convex hull
Export BBOX to GPKG
Exporting geolocation files including thumbnails: it could take some time !
Exporting metadata.csv !
####################
### 13. Enrich Provenance
####################
####################
### 3. Enrich Description
####################
INFO: Using glocal abstract
INFO: Using glocal purpose
INFO: Using glocal info
INFO: Using glocal edition
| └─ 20230524_REU-ermitage_UAV-02_3
|-------- └─ DCIM
|-------- └─ GPS
|---------------- └─ base_2023_05_24_pascal
|---------------- └─ reach_2023_05_24_drone
|------------------------ └─ reachsylvai_raw_202305240249_RINEX_3_03
|------------------------ └─ reachsylvai_raw_202305240330_RINEX_3_03
|---------------- └─ reach_2023_05_24_rover
|-------- └─ METADATA
|---------------- └─ tb
####################
### 14. Enrich Formats
####################
####################
### 15. Enrich data
####################
####################
#### END ####
####################
WARN: No metadata.txt in /home/sylvain/HDD5To/DATA/TELEMAC/20230802_REU-ermitage_UAV-02_1/METADATA to parse
INFO: GPS Folder found
Format id to obtain date, country name, optionnal place, and platform
####################
### 2. Enrich Title
####################
INFO: Using glocal title
Images drone, {platform}, {place}, Saint-Gilles, {country} - {date}
format field
####################
### 8. Enrich Language
####################
INFO: Using glocal language
####################
### 4. Enrich Subject
####################
INFO: Using glocal theme
INFO: Using glocal project
####################
### 5. Enrich Creator
####################
INFO: Using glocal author
INFO: Using glocal publisher
INFO: Using glocal owner
INFO: Using glocal pointOfContact
INFO: Using glocal processor
####################
### 11. Enrich Relation
####################
INFO: Using glocal thumbnail
INFO: Using glocal http
####################
### Reading DCIM folder
####################
Extract image coordinates and thumbnails: it could takes some time !
/home/sylvain/DroneMDT/enriched_func.py:14: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
enriched = df_global.iloc[0][keyword][0]
/home/sylvain/HDD5To/DATA/TELEMAC/20230802_REU-ermitage_UAV-02_1
Generate image with a set of images
####################
### 10. Enrich TemporalCoverage
####################
####################
### 00. Prepare Provenance (13) and Description (3)
####################
/home/sylvain/HDD5To/DATA/TELEMAC/20230802_REU-ermitage_UAV-02_1/DCIM/DJI_0338.JPG
####################
Fetching open-meteo
####################
Location: -21.082180100000002, 55.2220682
Begin: 2023-08-02 06-59-36 - End: 2023-08-02 07-16-47
Model: best_match
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1979 0 1979 0 0 7203 0 --:--:-- --:--:-- --:--:-- 7222
Convert Dataframe to UTM
Index(['FileName', 'GPSLatitude', 'GPSLongitude', 'GPSAltitude',
'FlightRollDegree', 'FlightPitchDegree', 'FlightYawDegree',
'GimbalRollDegree', 'GimbalPitchDegree', 'GimbalYawDegree',
'ImageWidth', 'ImageLength', 'BitsPerSample', 'ImageDescription',
'Make', 'Model', 'Orientation', 'SamplesPerPixel', 'XResolution',
'YResolution', 'ResolutionUnit', 'Software', 'DateTime',
'YCbCrPositioning', 'ExifOffset', 'GPSVersionID', 'ThumbnailImageWidth',
'ThumbnailImageLength', 'ThumbnailCompression', 'ThumbnailXResolution',
'ThumbnailYResolution', 'ThumbnailResolutionUnit',
'ThumbnailJPEGInterchangeFormat',
'ThumbnailJPEGInterchangeFormatLength', 'ExposureTime', 'FNumber',
'ExposureProgram', 'ISOSpeedRatings', 'ExifVersion', 'DateTimeOriginal',
'DateTimeDigitized', 'ComponentsConfiguration', 'ExposureBiasValue',
'MaxApertureValue', 'MeteringMode', 'LightSource', 'Flash',
'FocalLength', 'FlashPixVersion', 'ColorSpace', 'ExifImageWidth',
'ExifImageLength', 'FileSource', 'SceneType', 'ExposureMode',
'WhiteBalance', 'DigitalZoomRatio', 'FocalLengthIn35mmFilm',
'SceneCaptureType', 'GainControl', 'Contrast', 'Saturation',
'Sharpness', 'DeviceSettingDescription', 'BodySerialNumber',
'ThumbnailImage'],
dtype='object')
Compute convex hull
Export BBOX to GPKG
Exporting geolocation files including thumbnails: it could take some time !
Exporting metadata.csv !
####################
### 13. Enrich Provenance
####################
####################
### 3. Enrich Description
####################
INFO: Using glocal abstract
INFO: Using glocal purpose
INFO: Using glocal info
INFO: Using glocal edition
| └─ 20230802_REU-ermitage_UAV-02_1
|-------- └─ DCIM
|-------- └─ GPS
|-------- └─ METADATA
|---------------- └─ tb
####################
### 14. Enrich Formats
####################
####################
### 15. Enrich data
####################
####################
#### END ####
####################
WARN: No metadata.txt in /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_1/METADATA to parse
INFO: GPS Folder found
Format id to obtain date, country name, optionnal place, and platform
####################
### 2. Enrich Title
####################
INFO: Using glocal title
Images drone, {platform}, {place}, Saint-Gilles, {country} - {date}
format field
####################
### 8. Enrich Language
####################
INFO: Using glocal language
####################
### 4. Enrich Subject
####################
INFO: Using glocal theme
INFO: Using glocal project
####################
### 5. Enrich Creator
####################
INFO: Using glocal author
INFO: Using glocal publisher
INFO: Using glocal owner
INFO: Using glocal pointOfContact
INFO: Using glocal processor
####################
### 11. Enrich Relation
####################
INFO: Using glocal thumbnail
INFO: Using glocal http
####################
### Reading DCIM folder
####################
Extract image coordinates and thumbnails: it could takes some time !
/home/sylvain/DroneMDT/enriched_func.py:14: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
enriched = df_global.iloc[0][keyword][0]
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_1
Generate image with a set of images
####################
### 10. Enrich TemporalCoverage
####################
####################
### 00. Prepare Provenance (13) and Description (3)
####################
/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_1/DCIM/DJI_0175.JPG
####################
Fetching open-meteo
####################
Location: -21.082184599999998, 55.2219345
Begin: 2023-05-24 07-35-31 - End: 2023-05-24 07-58-33
Model: best_match
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1965 0 1965 0 0 7484 0 --:--:-- --:--:-- --:--:-- 7471
Convert Dataframe to UTM
Index(['FileName', 'GPSLatitude', 'GPSLongitude', 'GPSAltitude',
'FlightRollDegree', 'FlightPitchDegree', 'FlightYawDegree',
'GimbalRollDegree', 'GimbalPitchDegree', 'GimbalYawDegree',
'ImageWidth', 'ImageLength', 'BitsPerSample', 'ImageDescription',
'Make', 'Model', 'Orientation', 'SamplesPerPixel', 'XResolution',
'YResolution', 'ResolutionUnit', 'Software', 'DateTime',
'YCbCrPositioning', 'ExifOffset', 'GPSVersionID', 'ThumbnailImageWidth',
'ThumbnailImageLength', 'ThumbnailCompression', 'ThumbnailXResolution',
'ThumbnailYResolution', 'ThumbnailResolutionUnit',
'ThumbnailJPEGInterchangeFormat',
'ThumbnailJPEGInterchangeFormatLength', 'ExposureTime', 'FNumber',
'ExposureProgram', 'ISOSpeedRatings', 'ExifVersion', 'DateTimeOriginal',
'DateTimeDigitized', 'ComponentsConfiguration', 'ExposureBiasValue',
'MaxApertureValue', 'MeteringMode', 'LightSource', 'Flash',
'FocalLength', 'FlashPixVersion', 'ColorSpace', 'ExifImageWidth',
'ExifImageLength', 'FileSource', 'SceneType', 'ExposureMode',
'WhiteBalance', 'DigitalZoomRatio', 'FocalLengthIn35mmFilm',
'SceneCaptureType', 'GainControl', 'Contrast', 'Saturation',
'Sharpness', 'DeviceSettingDescription', 'BodySerialNumber',
'ThumbnailImage'],
dtype='object')
Compute convex hull
Export BBOX to GPKG
Exporting geolocation files including thumbnails: it could take some time !
Exporting metadata.csv !
####################
### 13. Enrich Provenance
####################
####################
### 3. Enrich Description
####################
INFO: Using glocal abstract
INFO: Using glocal purpose
INFO: Using glocal info
INFO: Using glocal edition
| └─ 20230524_REU-ermitage_UAV-02_1
|-------- └─ DCIM
|-------- └─ GPS
|---------------- └─ base_2023_05_24_pascal
|---------------- └─ reach_2023_05_24_drone
|------------------------ └─ reachsylvai_raw_202305240249_RINEX_3_03
|------------------------ └─ reachsylvai_raw_202305240330_RINEX_3_03
|---------------- └─ reach_2023_05_24_rover
|-------- └─ METADATA
|---------------- └─ tb
####################
### 14. Enrich Formats
####################
####################
### 15. Enrich data
####################
####################
#### END ####
####################
[14]:
# Set prefix for generated json and csv entities
out_prefix = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
out_prefix
[14]:
'20231022-152725'
[18]:
# Visualize output
df.head()
[18]:
| Identifier | Title | Description | Subject | Creator | Date | Type | Language | SpatialCoverage | TemporalCoverage | Relation | Rights | Provenance | Format | Data | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 20230524_REU-ermitage_UAV-02_4 | title:Images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230524 - 02_4 | abstract:"Jeu de données composé d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermita... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | 2023-05-24 09-53-20 - 2023-05-24 10-03-57 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/jpg_\ndistribution:application/geopackage+sqlite3[GeoPackage] | source:SurveyMetadata.gpkg@/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4/GPS/Surve... | |
| 1 | 20230524_REU-ermitage_UAV-02_2 | title:Images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230524 - 02_2 | abstract:"Jeu de données composé d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermita... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | 2023-05-24 09-05-25 - 2023-05-24 09-26-24 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/jpg_\ndistribution:application/geopackage+sqlite3[GeoPackage] | source:SurveyMetadata.gpkg@/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2/GPS/Surve... | |
| 2 | 20230705_REU-ermitage_UAV-02_1 | title:Images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230705 - 02_1 | abstract:"Jeu de données composé d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermita... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | 2023-07-05 10-33-06 - 2023-07-05 10-48-40 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/jpg_\ndistribution:application/geopackage+sqlite3[GeoPackage] | source:SurveyMetadata.gpkg@/home/sylvain/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1/GPS/Surve... | |
| 3 | 20230524_REU-ermitage_UAV-02_3 | title:Images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230524 - 02_3 | abstract:"Jeu de données composé d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermita... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | 2023-05-24 09-33-18 - 2023-05-24 09-49-53 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/jpg_\ndistribution:application/geopackage+sqlite3[GeoPackage] | source:SurveyMetadata.gpkg@/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_3/GPS/Surve... | |
| 4 | 20230802_REU-ermitage_UAV-02_1 | title:Images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230802 - 02_1 | abstract:"Jeu de données composé d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermita... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | 2023-08-02 06-59-36 - 2023-08-02 07-16-47 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/jpg_\ndistribution:application/geopackage+sqlite3[GeoPackage] | source:SurveyMetadata.gpkg@/home/sylvain/HDD5To/DATA/TELEMAC/20230802_REU-ermitage_UAV-02_1/GPS/Surve... |
[19]:
# Visualize Description
print(df.iloc[0]['Description'])
abstract:"Jeu de données composé d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermitage, Saint-Gilles, Réunion à la date suivante : 20230524.
<br />Les vols ont été réalisés dans le but de créer des modèles numériques d'élévations pour cartographier la rugosité récifale du lagon jusqu'à la pente externe. Les données seront utilisées dans le cadre du projet TELEMAC.
<br />
<br /><b>Le dépôt est composé des éléments suivants:</b>
<br /> - 00_: Planche d'aperçu des images
<br /> - DCIM.zip: Images brutes issues du drone
<br /> - GPS.zip: Geopackage contenant l’emprise du survol ainsi que la géolocalisation des images accompagnées de leurs miniatures dans la table d’attribut en base64. RINEX et LLH issus d'un Emlid Reach M2 synchronisé avec la LED de navigation (événement envoyé dans le log du récepteur GNSS lors du déclenchement d'une image). Le but est de pouvoir réaliser un PPK (similaire au RTK en post-traitement) et ainsi disposer d'une position centrimétrique sur chaque image.
<br /> - METADATA.zip: Métadonnées au format ISO19115, Rapports avec miniatures des images de drone (dossier tb) et statistiques de vols.
<br />
<br /><b>Arborescence d'origine:</b>
<br />│ └─ 20230524_REU-ermitage_UAV-02_4
<br />│-------- └─ DCIM
<br />│-------- └─ GPS
<br />│---------------- └─ base_2023_05_24_pascal
<br />│---------------- └─ reach_2023_05_24_drone
<br />│------------------------ └─ reachsylvai_raw_202305240249_RINEX_3_03
<br />│------------------------ └─ reachsylvai_raw_202305240330_RINEX_3_03
<br />│---------------- └─ reach_2023_05_24_rover
<br />│-------- └─ METADATA
<br />│---------------- └─ tb
<br />
<br /><b>Informations de survol:</b>
<br />- Camera model and parameters:
<br /> Make: Hasselblad
<br /> Model: L1D-20c
<br /> Width: 5472
<br /> Height: 3648
<br /> Focal: 28
<br /> WhiteBalance: Manual
<br /> ExposureMode: Auto Exposure
<br /> ColoSpace: sRGB
<br /> EV: -0.7
<br /> MeteringMode: CenterWeightedAverage
<br /> Camera Pitch: -70.00
<br />
<br />- Survey informations:
<br /> No Images: 193
<br /> Median height: 70 meters
<br /> Survey area: 4.16 hectares
<br /> Survey from: 2023:05:24 09:53:20 to: 2023:05:24 10:03:57
<br />
<br />D'autres versions plus récentes de ce dépôt sont disponibles. Ces dernières comprennent notamment les données traitées avec OpenDroneMap"_
purpose:" "_
info:"This study was funded by the European Regional Development Fund (ERDF) within the programme Interreg V 2014-2020 through the project G2OI"_
edition:"RAW DATA"
[20]:
##############
#### ISO 19115 config
##############
## Export to csv
df.to_csv(os.path.join(output, out_prefix + "_iso19115-metadata.csv"), quoting=1, quotechar='"', index=False)
## Generate json
iso19115config = {'profile': {'id': 'iso19115',
'name': 'iso19115',
'project': 'Generate iso19115 with Geoflow',
'organization': 'IRD',
'environment': {'file': '.env',
'hide_env_vars': ['MOTDEPASSE']},
'logos': ['https://en.ird.fr/sites/ird_fr/files/2019-08/logo_IRD_2016_BLOC_UK_COUL.png'],
'mode': 'entity'},
'metadata': {'entities':
[{'handler': 'csv',
'source': os.path.join(output, out_prefix) + "_iso19115-metadata.csv"}],
'contacts':
[{'handler': 'csv',
'source': 'https://drive.ird.fr/s/EYS3qccyB28PrA9/download/geoflow_g2oi_contacts.csv'}]},
'software': [],
'actions':
[{'id': 'geometa-create-iso-19115',
'options' :
{"logo": True},
"run": True}]}
# Export json to file
with open(os.path.join(output, out_prefix + '_iso19115-config.json'), 'w') as f:
json.dump(iso19115config, f, indent=4)
[9]:
# Optional : could start from here => reload dataframe from csv
# df = {}
# df = pd.read_csv(output, out_prefix + "_iso19115-metadata.csv")
[21]:
### Execute geoflow
xml19115 = run_geoflow(os.path.join(output, out_prefix + '_iso19115-config.json'), geoflow_folder)
os.chdir(dronemdt_folder)
R[write to console]: Rows: 129 Columns: 14
R[write to console]: ── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (14): Identifier, Email, OrganizationName, PositionName, LastName, First...
R[write to console]:
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
R[write to console]: Rows: 6 Columns: 15
R[write to console]: ── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (15): Identifier, Title, Description, Subject, Creator, Date, Type, Lang...
R[write to console]:
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
R[write to console]: Loading required package: geometa
R[write to console]: Loading required package: ows4R
[22]:
# Geoflow job log folder
print(xml19115[0])
/home/sylvain/geoflow-g2oi/jobs/20231022153101
[23]:
## Copy generated ISO19115 to original metadata folder
for id in range(len(df)):
jobs19115 = xml19115[0] + '/entities/' + df.iloc[id]['Identifier'] + '/metadata/'
fname19115 = df.iloc[id]['Identifier'] + '_ISO-19115.xml'
if fname19115 in os.listdir(jobs19115):
shutil.copyfile(jobs19115+fname19115,
input + "/" + df.iloc[id]['Identifier'] + '/METADATA/' + fname19115)
Part 3. Upload RAW data to Zenodo
ZIP all folders and reference files in dataframe (data column)
[24]:
### Zip all folders
zip_each_folder(input, df)
### Reference Zip's in Dataframe
for zip in range(len(df)):
source = []
sess_path = os.path.join(input, df.iloc[zip]['Identifier'])
source = os.listdir(sess_path)
if len(source) < 1:
pass
else:
source_file = []
for file in range(len(source)):
if "PROCESSED_DATA.zip" in source:
pass
else:
if os.path.isfile(os.path.join(sess_path, source[file])):
if len(source) < 2:
df.iloc[zip, df.columns.get_loc('Data')] = "source:" + source[file] + "@" + sess_path + "/" + source[file] + "_\n" + "sourceType:other_\nsourceZip:false"
source_file = source[file]
else:
source_file.append(source[file])
# df.iloc[zip, df.columns.get_loc('Data')] = "source:" + source[file] + "@" + sess_path + source[file] + "_\n" + "sourceType:other_\nsourceZip:false"
if ".zip" in df.iloc[zip, df.columns.get_loc('Data')]:
pass
else:
df.iloc[zip, df.columns.get_loc('Data')] = "source:"
for i in range(len(source_file)):
df.iloc[zip, df.columns.get_loc('Data')] = str(df.iloc[zip, df.columns.get_loc('Data')]) + source_file[i] + "@" + sess_path + "/" + source_file[i] + ","
if "," in str(df.iloc[zip, df.columns.get_loc('Data')])[-1]:
df.iloc[zip, df.columns.get_loc('Data')] = str(df.iloc[zip, df.columns.get_loc('Data')])[:-1] + "_\n" + "sourceType:other_\nupload:true"
INFO: Ziping 20230524_REU-ermitage_UAV-02_4
INFO: 00_sample_rawdata_overview.png is not a folder
INFO: Ziping 20230524_REU-ermitage_UAV-02_2
INFO: 00_sample_rawdata_overview.png is not a folder
INFO: Ziping 20230705_REU-ermitage_UAV-02_1
INFO: 00_sample_rawdata_overview.png is not a folder
INFO: Ziping 20230524_REU-ermitage_UAV-02_3
INFO: 00_sample_rawdata_overview.png is not a folder
INFO: Ziping 20230802_REU-ermitage_UAV-02_1
INFO: 00_sample_rawdata_overview.png is not a folder
INFO: Ziping 20230524_REU-ermitage_UAV-02_1
INFO: 00_sample_rawdata_overview.png is not a folder
Generate config files for geoflow or re-use dataframe later
[25]:
### Export raw data df to csv
fname = out_prefix + '_zenodo-rawdata'
df.to_csv(os.path.join(output, fname + ".csv"), quoting=1, quotechar='"', index=False)
### Configure and export Zenodo json
# SANDBOX
# zen_json = zenodo_json(fname, 'sandbox', output, out_prefix)
# PRODUCTION
zen_json = zenodo_json(fname, 'no_sandbox', output, out_prefix)
with open(os.path.join(output, out_prefix + '_rawfiles-config.json'), 'w') as f:
json.dump(zen_json, f, indent=4)
Execute Geoflow to upload to Zenodo
/! DISABLED SINCE FRIDAY 2023 10 13 /!
/! Zenodo upgrade /!
[26]:
# ### Execute geoflow :: upload to geoflow
# # upload_raw_to_zenodo =
# config_raw = os.path.join(output, out_prefix + '_rawfiles-config.json')
# # run_geoflow(config_raw, geoflow_folder)
# os.chdir(geoflow_folder)
# # subprocess.run(["R", "-e", "geoflow::initWorkflow('{}')".format(config_raw)], capture_output=False)
# subprocess.call(["R", "-e", "geoflow::executeWorkflow('{}')".format(config_raw)],
# stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT,)
# os.chdir(dronemdt_folder)
Upload to Zenodo with Python
[212]:
# %load_ext autoreload
# %autoreload 2
# del(zenul)
# from zenodo_helper import *
# help(zenul)
data_zip
[212]:
['000_photogrammetry_report.pdf', 'PROCESSED_DATA.zip']
[97]:
### Upload to Zenodo
print("#### Upload zip files to Zenodo")
base_url = "https://zenodo.org/api/"
for zipul in range(len(df)):
zenodo_baseurl = base_url
data_zip = df.iloc[zipul]['Data'].split('source:')[1].split('_\n')[0]
if "," in data_zip:
data_ziptemp = data_zip.split(',')
data_zip = []
for dt in range(len(data_ziptemp)):
data_zip.append(data_ziptemp[dt].split('@')[0])
else:
data_zip.split('@')[0]
# data_zip = source_file
print(data_zip)
print("Initialize deposit")
r = check_token(zenodo_baseurl, ACCESS_TOKEN)
zenval = zenvar(r)
print("prereserved doi:"+zenval[1])
print("Write DOI to dataframe")
if 'id:' in df.iloc[zipul]['Identifier']:
pass
else:
df.iloc[zipul, df.columns.get_loc('Identifier')] = "id:" + df.iloc[zipul]['Identifier'] + "_\ndoi:" + zenval[1]
df.iloc[zipul, df.columns.get_loc("Provenance")] = df.iloc[zipul]["Provenance"] + "_\nprocess:Raw dataset uploaded to " + base_url.split('api')[0] + "record/" + str(zenval[2])
print("Enrich upload with metadata")
zen_metadata = zenmdt(zenodo_baseurl, ACCESS_TOKEN, zenval[2], df, zipul)
# zen_metadata.text
print("upload data")
print("Trying upload number: 1")
for file in data_zip:
file_list = [file]
zen_upload = zenul(zenval[0], ACCESS_TOKEN, os.path.join(input, df.iloc[zipul]['Identifier'].split('_\n')[0].split(':')[1]), file_list)
# zen_upload = zenul(zenvar(newv), ACCESS_TOKEN, os.getcwd(), test_list)
# zen_upload = zenul2(zenodo_baseurl, str(zenval[-1]), ACCESS_TOKEN, os.path.join(input, df.iloc[zipul]['Identifier'].split('_\n')[0].split(':')[1]), data_zip)
print("Upload status OK? " + str(zen_upload.ok))
print("Control files: first pass")
ul_count = 1
while zen_upload.status_code != 201:
if zen_upload.status_code == 404:
print("Version doesn't exists ! Please check your record_id")
break
else:
ul_count += 1
print("Retry number: " + str(ul_count))
zen_upload = zenul(zenval[0], ACCESS_TOKEN, os.path.join(input, df.iloc[zipul]['Identifier'].split('_\n')[0].split(':')[1]), file_list)
if zen_upload.status_code == 403:
print("permission denied!")
time.sleep(10)
### Too agressive
# print("Control files: second pass")
# if zen_upload.status_code == 201:
# # files_ul = zenfiles(zenodo_baseurl, ACCESS_TOKEN, str(zenval[2]))
# while len(zenfiles(zenodo_baseurl, ACCESS_TOKEN, str(zenval[2])).json()) != len(data_zip):
# if zen_upload.status_code == 404:
# print("Version doesn't exists ! Please check your record_id")
# break
# else:
# ul_count += 1
# print("Retry number: " + str(ul_count))
# zen_upload = zenul(zenval[0], ACCESS_TOKEN, os.path.join(input, df.iloc[zipul]['Identifier'].split('_\n')[0].split(':')[1]), data_zip)
# # files_ul = zenfiles(zenodo_baseurl, ACCESS_TOKEN, str(zenval[2]))
### Publish to Zenodo
# print("Publish record")
# zenpublish(zenodo_baseurl, ACCESS_TOKEN, zenval[2])
#### Upload zip files to Zenodo
Initialize deposit
Allowed to deposit some files
prereserved doi:10.5281/zenodo.10037633
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
Sleep 1 second before new upload
upload: GPS.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: GPS.zip
Sleep 1 second before new upload
upload: 00_sample_rawdata_overview.png
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: 00_sample_rawdata_overview.png
permission denied!
Retry number: 3
Sleep 1 second before new upload
upload: 00_sample_rawdata_overview.png
Sleep 1 second before new upload
upload: DCIM.zip
Upload status OK? True
Control files: first pass
Sleep 1 second before new upload
upload: METADATA.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: METADATA.zip
Initialize deposit
Allowed to deposit some files
prereserved doi:10.5281/zenodo.10037635
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
Sleep 1 second before new upload
upload: GPS.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: GPS.zip
permission denied!
Retry number: 3
Sleep 1 second before new upload
upload: GPS.zip
permission denied!
Retry number: 4
Sleep 1 second before new upload
upload: GPS.zip
Sleep 1 second before new upload
upload: 00_sample_rawdata_overview.png
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: 00_sample_rawdata_overview.png
Sleep 1 second before new upload
upload: DCIM.zip
Upload status OK? True
Control files: first pass
Sleep 1 second before new upload
upload: METADATA.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: METADATA.zip
Initialize deposit
Allowed to deposit some files
prereserved doi:10.5281/zenodo.10037640
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
Sleep 1 second before new upload
upload: GPS.zip
Upload status OK? True
Control files: first pass
Sleep 1 second before new upload
upload: 00_sample_rawdata_overview.png
Upload status OK? True
Control files: first pass
Sleep 1 second before new upload
upload: DCIM.zip
Upload status OK? True
Control files: first pass
Sleep 1 second before new upload
upload: METADATA.zip
Upload status OK? True
Control files: first pass
Initialize deposit
Allowed to deposit some files
prereserved doi:10.5281/zenodo.10037645
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
Sleep 1 second before new upload
upload: GPS.zip
Upload status OK? True
Control files: first pass
Sleep 1 second before new upload
upload: 00_sample_rawdata_overview.png
Upload status OK? True
Control files: first pass
Sleep 1 second before new upload
upload: DCIM.zip
Upload status OK? True
Control files: first pass
Sleep 1 second before new upload
upload: METADATA.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: METADATA.zip
Initialize deposit
Allowed to deposit some files
prereserved doi:10.5281/zenodo.10037649
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
Sleep 1 second before new upload
upload: GPS.zip
Upload status OK? True
Control files: first pass
Sleep 1 second before new upload
upload: 00_sample_rawdata_overview.png
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: 00_sample_rawdata_overview.png
Sleep 1 second before new upload
upload: DCIM.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: DCIM.zip
permission denied!
Retry number: 3
Sleep 1 second before new upload
upload: DCIM.zip
Sleep 1 second before new upload
upload: METADATA.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: METADATA.zip
permission denied!
Retry number: 3
Sleep 1 second before new upload
upload: METADATA.zip
Initialize deposit
Allowed to deposit some files
prereserved doi:10.5281/zenodo.10037659
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
Sleep 1 second before new upload
upload: GPS.zip
Upload status OK? True
Control files: first pass
Sleep 1 second before new upload
upload: 00_sample_rawdata_overview.png
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: 00_sample_rawdata_overview.png
Sleep 1 second before new upload
upload: DCIM.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: DCIM.zip
permission denied!
Retry number: 3
Sleep 1 second before new upload
upload: DCIM.zip
Sleep 1 second before new upload
upload: METADATA.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: METADATA.zip
permission denied!
Retry number: 3
Sleep 1 second before new upload
upload: METADATA.zip
permission denied!
Retry number: 4
Sleep 1 second before new upload
upload: METADATA.zip
[205]:
### Check DOI's
for doi in range(len(df)):
print(df.iloc[doi]["Identifier"])
id:20230524_REU-ermitage_UAV-02_4_
doi:10.5281/zenodo.10037633
id:20230524_REU-ermitage_UAV-02_2_
doi:10.5281/zenodo.10037635
id:20230705_REU-ermitage_UAV-02_1_
doi:10.5281/zenodo.10037640
id:20230524_REU-ermitage_UAV-02_3_
doi:10.5281/zenodo.10037645
id:20230802_REU-ermitage_UAV-02_1_
doi:10.5281/zenodo.10037649
id:20230524_REU-ermitage_UAV-02_1_
doi:10.5281/zenodo.10037659
[123]:
### Export Dataframe to csv
# ### Export Dataframe to csv
csv_published = out_prefix + "_zenodo-rawdata"
df.to_csv(os.path.join(output, csv_published + ".csv"), quoting=1, quotechar='"', index=False)
# ### Export geoflow config to json
if sandbox == True:
zenodo_url = "https://sandbox.zenodo.org/api"
else:
zenodo_url = "https://zenodo.org/api"
rawdata_config = {'profile': {'id': 'rawdata', 'name': 'rawdata', 'project': 'Publish rawdata with Geoflow', 'organization': 'IRD', 'environment': {'file': '.env', 'hide_env_vars': ['MOTDEPASSE']}, 'logos': ['https://en.ird.fr/sites/ird_fr/files/2019-08/logo_IRD_2016_BLOC_UK_COUL.png'], 'mode': 'entity'}, 'metadata': {'entities': [{'handler': 'csv', 'source': os.path.join(output, out_prefix) + "_zenodo-rawdata.csv"}], 'contacts': [{'handler': 'csv', 'source': 'https://drive.ird.fr/s/EYS3qccyB28PrA9/download/geoflow_g2oi_contacts.csv'}]}, 'software': [{"id": "my-zenodo","type": "output","software_type": "zenodo","parameters": {"url": zenodo_url,"token": "{{ ZENODO_TOKEN }}","logger": "DEBUG"},"properties": {"clean": {"run": False}}}], 'actions': [{"id": "zen4R-deposit-record","options": {"update_files": True,"communities": "uav","depositWithFiles": True,"publish": False,"update_metadata": True,"strategy": "newversion","deleteOldFiles": True},"run": True}]}
with open(os.path.join(output, out_prefix + '_zenodo-rawdata-config.json'), 'w') as f:
json.dump(rawdata_config, f, indent=4)
Part 4. Download from ODM
[174]:
# Take csv generated from python's zenodo upload as updated dataframe
df = {}
df = pd.read_csv(os.path.join(input, out_prefix + '_zenodo-rawdata.csv'))
df.head()
[174]:
| Identifier | Title | Description | Subject | Creator | Date | Type | Language | SpatialCoverage | TemporalCoverage | Relation | Rights | Provenance | Format | Data | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | id:20230524_REU-ermitage_UAV-02_4_\ndoi:10.5281/zenodo.10037633 | title:Images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230524 - 02_4 | abstract:"Jeu de données composé d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermita... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | NaN | 2023-05-24 09-53-20 - 2023-05-24 10-03-57 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/jpg_\ndistribution:application/geopackage+sqlite3[GeoPackage] | source:GPS.zip@/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4/GPS.zip,00_sample_raw... |
| 1 | id:20230524_REU-ermitage_UAV-02_2_\ndoi:10.5281/zenodo.10037635 | title:Images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230524 - 02_2 | abstract:"Jeu de données composé d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermita... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | NaN | 2023-05-24 09-05-25 - 2023-05-24 09-26-24 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/jpg_\ndistribution:application/geopackage+sqlite3[GeoPackage] | source:GPS.zip@/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2/GPS.zip,00_sample_raw... |
| 2 | id:20230705_REU-ermitage_UAV-02_1_\ndoi:10.5281/zenodo.10037640 | title:Images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230705 - 02_1 | abstract:"Jeu de données composé d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermita... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | NaN | 2023-07-05 10-33-06 - 2023-07-05 10-48-40 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/jpg_\ndistribution:application/geopackage+sqlite3[GeoPackage] | source:GPS.zip@/home/sylvain/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1/GPS.zip,00_sample_raw... |
| 3 | id:20230524_REU-ermitage_UAV-02_3_\ndoi:10.5281/zenodo.10037645 | title:Images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230524 - 02_3 | abstract:"Jeu de données composé d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermita... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | NaN | 2023-05-24 09-33-18 - 2023-05-24 09-49-53 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/jpg_\ndistribution:application/geopackage+sqlite3[GeoPackage] | source:GPS.zip@/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_3/GPS.zip,00_sample_raw... |
| 4 | id:20230802_REU-ermitage_UAV-02_1_\ndoi:10.5281/zenodo.10037649 | title:Images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230802 - 02_1 | abstract:"Jeu de données composé d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermita... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | NaN | 2023-08-02 06-59-36 - 2023-08-02 07-16-47 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/jpg_\ndistribution:application/geopackage+sqlite3[GeoPackage] | source:GPS.zip@/home/sylvain/HDD5To/DATA/TELEMAC/20230802_REU-ermitage_UAV-02_1/GPS.zip,00_sample_raw... |
[125]:
### SEARCH AND DOWNLOAD FROM ODM
odm_projects = list_all_projects()
for projects in range(len(odm_projects)):
for p in range(len(df)):
if df.iloc[p]['Identifier'].split('_\n')[0].split(':')[1] in odm_projects[projects]['name']:
print(projects, odm_projects[projects]['id'], odm_projects[projects]['name'], odm_projects[projects]['tasks'])
project_id = odm_projects[projects]['id']
tasks = odm_projects[projects]['tasks']
res = requests.post(settings.SERVER + '/api/token-auth/',
data={'username': settings.USERNAME,
'password': settings.PASSWORD}).json()
if 'token' in res:
print("Still logged-in!")
token = res['token']
for task in range(len(tasks)):
task_id = tasks[task]
print(task_id)
res = requests.get(settings.SERVER + "/api/projects/{}/tasks/{}/download/all.zip".format(project_id, task_id),
headers={'Authorization': 'JWT {}'.format(token)},
stream=True)
# token = res['token']
sess_path = os.path.join(input, df.iloc[p]['Identifier'].split('_\n')[0].split(':')[1])
### Reading PROCESSED_DATA folder
### Check and create PROCESSED_DATA Directory if it not exists
# process_dir=os.path.join(sess_path, 'PROCESSED_DATA')
# if os.path.exists(process_dir):
if os.path.exists(sess_path):
if 'PROCESSED_DATA.zip' in os.listdir(sess_path):
print("INFO: data already found in {} !".format(df.iloc[p]['Identifier'].split('_\n')[0].split(':')[1]))
else:
# print("INFO: Create PROCESSED_DATA folder in {} folder")
# os.mkdir(process_dir)
print("Downloading all.zip")
with open(os.path.join(input, df.iloc[p]['Identifier'].split('_\n')[0].split(':')[1], "all.zip"), 'wb') as f:
for chunk in res.iter_content(chunk_size=1024):
if chunk:
f.write(chunk)
# file_newname = "ortho_" + df.iloc[p]['Identifier'].split('_\n')[0].split(':')[1] + ".tif"
file_newname = 'PROCESSED_DATA.zip'
print("Renaming all.zip to PROCESSED_DATA.zip")
os.rename(os.path.join(sess_path, "all.zip"), os.path.join(sess_path, file_newname))
print("Saved " + file_newname)
res = requests.get(settings.SERVER + "/api/projects/{}/tasks/{}/download/report.pdf".format(project_id, task_id),
headers={'Authorization': 'JWT {}'.format(token)},
stream=True)
pdf_newname = '000_photogrammetry_report.pdf'
if pdf_newname in os.listdir(sess_path):
print("INFO: PDF already found in {} !".format(df.iloc[p]['Identifier'].split('_\n')[0].split(':')[1]))
else:
print("Downloading report.pdf")
with open(os.path.join(input, df.iloc[p]['Identifier'].split('_\n')[0].split(':')[1], "report.pdf"), 'wb') as pdf:
for chunk in res.iter_content(chunk_size=1024):
if chunk:
pdf.write(chunk)
os.rename(os.path.join(sess_path, "report.pdf"), os.path.join(sess_path, pdf_newname))
print("Saved " + pdf_newname)
### Enrich Metadata
dfdlodm = df
dfdlodm.iloc[p, dfdlodm.columns.get_loc('Provenance')] = dfdlodm.iloc[p]["Provenance"] + "_\nprocess:Download from WebODM"
dfdlodm.iloc[p, dfdlodm.columns.get_loc('Format')] = "resource:image/tiff_\ndistribution:image/tiff[GeoTIFF]"
dfdlodm.iloc[p, dfdlodm.columns.get_loc('Data')] = pdf_newname + "@"+ os.path.join(sess_path, pdf_newname) + "," + file_newname + "@"+ os.path.join(sess_path, file_newname) + "_\nsourceType:other_\nsourceZip:False"
Successfully Logged-in!
########
id name permissions
----------
160 PyWebODM - 20221024_SYC-aldabra-passe-arm01_UAV-02_28
159 PyWebODM - 20221021_SYC-aldabra-arm01_UAV-02_10
156 PyWebODM - 20221024_SYC-aldabra-gigi_UAV-02_31
155 PyWebODM - 20221024_SYC-aldabra-gigi_UAV-02_30
154 PyWebODM - 20221024_SYC-aldabra-gigi_UAV-02_29
153 PyWebODM - 20221023_SYC-aldabra-arm06_UAV-02_21
152 PyWebODM - 20221023_SYC-aldabra-arm06_UAV-02_20
151 PyWebODM - 20221023_SYC-aldabra-arm06_UAV-02_19
150 PyWebODM - 20221023_SYC-aldabra-arm06_UAV-02_18
149 PyWebODM - 20221023_SYC-aldabra-arm06_UAV-02_17
148 PyWebODM - 20221022_SYC-aldabra-passe-dubois_UAV-02_14
147 PyWebODM - 20221022_SYC-aldabra-passe-dubois_UAV-02_13
146 PyWebODM - 20221022_SYC-aldabra-passe-dubois_UAV-02_12
144 PyWebODM - 20221022_SYC-aldabra-arm06_UAV-02_16
143 PyWebODM - 20221022_SYC-aldabra-arm06_UAV-02_15
142 PyWebODM - 20221021_SYC-aldabra-arm01_UAV-02_9
141 PyWebODM - 20221021_SYC-aldabra-arm01_UAV-02_7
139 PyWebODM - 20221025_SYC-aldabra-arm01_UAV-02_33
122 PyWebODM - 20221024_SYC-aldabra-passe-dubois-MSP_UAV-02_27
121 PyWebODM - 20221024_SYC-aldabra-passe-dubois-40m_UAV-02_26
120 PyWebODM - 20221024_SYC-aldabra-passe-dubois-40m_UAV-02_25
119 PyWebODM - 20221024_SYC-aldabra-passe-dubois-40m_UAV-02_24
118 PyWebODM - 20221024_SYC-aldabra-passe-dubois-40m_UAV-02_23
101 PyWebODM - 20221021_SYC-aldabra-arm01_UAV-02_8
98 PyWebODM - 20220920_SYC-aldabra-arm01_UAV-02_6
97 PyWebODM - 20220920_SYC-aldabra-arm01_UAV-02_5
96 PyWebODM - 20220920_SYC-aldabra-arm01_UAV-02_4
95 PyWebODM - 20220920_SYC-aldabra-arm01_UAV-02_3
94 PyWebODM - 20220920_SYC-aldabra-arm01_UAV-02_2
93 PyWebODM - 20220920_SYC-aldabra-arm01_UAV-02_1
92 PyWebODM - 20220919_SYC-aldabra-arm01_UAV-02_1
91 yann
90 First Project
89 20230626_COM-mitsamiouli_UAV-01_2
88 20230628_COM-maloudja_UAV-02_2
87 20230626_COM-bandamaji_UAV-01_1
86 20230626_COM-bandamaji_UAV-02_1
85 20230626_COM-bangoua-kouni_UAV-01_1
84 20230627_COM-vouvouni_UAV-01_2
83 20230626_COM-mitsamiouli_UAV-02_1
82 20230627_COM-vouvouni_UAV-02_1
81 20230627_COM-vouvouni_UAV-01_1
80 20230701_COM-nioumadzaha-mitsoudje_UAV-01_2
79 20230627_COM-vouvouni_UAV-02_2
78 20230701_COM-nioumadzaha-mitsoudje_UAV-01_1
77 20230626_COM-mitsamiouli_UAV-01_1
76 20230701_COM-nioumadzaha-mitsoudje_UAV-01_3
75 20230626_COM-bandamaji_UAV-01_2
74 20230701_COM-nioumadzaha-mitsoudje_UAV-02_1
73 20230626_COM-bangoua-kouni_UAV-02_1
72 Premier projet
71 PyWebODM - 20230917_MDG-nosy-fanhy_UAV_02
69 20230917_MDG-nosy-fanhy_UAV_02_4
68 20230917_MDG-nosy-fanhy_UAV_02_3
67 20230917_MDG-nosy-fanhy_UAV_02_2
66 20230917_MDG-nosy-fanhy_UAV_02_1
64 20230915_MDG-nosy-sakatia_UAV-02_2
63 20230915_MDG-nosy-sakatia_UAV-02_1
62 20230912_MDG-nosy-sakatia_UAV-02_1
59 aldabra_session_20221021_6187images
58 20230524_REU-ermitage_UAV-02_4
57 PyWebODM - 2023_05_01_Anakao_Prince_Anakao_drone_01_7
56 PyWebODM - 2023_05_01_Anakao_Prince_Anakao_drone_01_6
55 PyWebODM - 2023_05_07_Mangily_Massif_Roses_drone_01_2
54 PyWebODM - 2023_05_07_Mangily_Massif_Roses_drone_01_3
47 PyWebODM - 2023_05_05_Madagascar_Sarodrano_village_01_1
45 PyWebODM - 2023_05_05_Madagascar_Sarodrano_Reef_Manjaka_Gauche_drone_01_14
42 PyWebODM - 2023_05_05_Madagascar_Sarodrano_Reef_Manjaka_Gauche_drone_01_14
39 PyWebODM - 2023_05_04_Saint_Augustin_pirogues_drone_01_2
38 PyWebODM - 2023_05_04_Nosy_Velomitahy_drone_01_6
35 PyWebODM - 2023_05_07_Mangily_estran_drone_01_4
34 PyWebODM - 2023_05_07_Mangily_estran_drone_01_3
33 PyWebODM - 2023_05_06_Mangily_estran_drone_01_2
32 PyWebODM - 2023_05_06_Mangily_estran_drone_01_1
30 PyWebODM - 2023_05_05_Madagascar_Sarodrano_Reef_Manjaka_Gauche_drone_01_15
29 PyWebODM - 2023_05_05_Madagascar_Sarodrano_Reef_Manjaka_Gauche_drone_01_14
28 PyWebODM - 2023_05_05_GRT_fausse_passe_drone_01_8
27 PyWebODM - 2023_05_05_GRT_fausse_passe_drone_01_7
26 PyWebODM - 2023_05_04_Saint_Augustin_pirogues_drone_01_2
25 PyWebODM - 2023_05_04_Nosy_Velomitahy_drone_01_6
24 PyWebODM - 2023_05_04_GRT_micro_atoll_zone_B_drone_01_2
23 PyWebODM - 2023_05_04_GRT_micro_atoll_zone_B_drone_01_1
22 PyWebODM - 2023_05_03_Sarodrano_IOT_drone_01_9
21 PyWebODM - 2023_05_03_Sarodrano_IOT_drone_01_11
20 PyWebODM - 2023_05_03_sarodrano_IHSM_Belaza_drone_01_13
19 PyWebODM - 2023_05_03_Nosy_Veolmitahy_drone_01_2
18 PyWebODM - 2023_05_02_Sarodrano_pirogues_drone_01_7
17 PyWebODM - 2023_05_02_Sarodrano_IOT_drone_01_6
13 PyWebODM - 2023_05_01_Anakao_Prince_Anakao_drone_01_4
12 PyWebODM - 2023_05_01_Anakao_Prince_Anakao_drone_01_3
11 PyWebODM - 2023_05_01_Anakao_Prince_Anakao_drone_01_2
9 PyWebODM - 2023_04_30_Nosy_Ve_drone_01_5
7 PyWebODM - 2023_04_30_Nosy_Ve_drone_01_2
6 PyWebODM - 2023_04_30_Nosy_Ve_drone_01_1
5 PyWebODM - 2023_04_29_Anakao_Prince_Anakao_drone_01_1
440 PyWebODM - uav_2023_05_07_madagascar-massif-des-roses_Mavic2Pro_02_2
439 PyWebODM - uav_2023_05_07_madagascar-massif-des-roses_Mavic2Pro_02_1
438 PyWebODM - uav_2023_05_07_madagascar-ifaty_Mavic2Pro_02_1
437 PyWebODM - uav_2023_05_05_madagascar-sarodrano_Mavic2Pro_02_2
436 PyWebODM - uav_2023_05_05_madagascar-sarodrano_Mavic2Pro_02_1
435 PyWebODM - uav_2023_05_04_madagascar-saintaugustin_Mavic2Pro_02_1
434 PyWebODM - uav_2023_05_04_madagascar-atoll_Mavic2Pro_02_1
433 PyWebODM - uav_2023_05_03_madagascar-iot_Mavic2Pro_02_1
432 PyWebODM - uav_2023_05_02_madagascar-sarodrano_Mavic2Pro_02_1
431 PyWebODM - uav_2023_05_02_madagascar-iot_Mavic2Pro_02_1
430 PyWebODM - uav_2023_05_02_madagascar-anakao_Mavic2Pro_02_1
428 PyWebODM - uav_2023_04_30_madagascar-nosyve_Mavic2Pro_02_3
427 PyWebODM - uav_2023_04_30_madagascar-nosyve_Mavic2Pro_02_2
426 PyWebODM - uav_2023_04_30_madagascar-nosyve_Mavic2Pro_02_1
424 PyWebODM - uav_2023_04_27_madagascar-tulear_Mavic2Pro_02_1
423 20230802_REU-ermitage_UAV-02_1
422 20230705_REU-ermitage_UAV-02_1
420 20230524_REU-ermitage_UAV-02_3
419 20230524_REU-ermitage_UAV-02_2
412 PyWebODM - gp3 -plancha
411 PyWebODM - gp2 -plancha
410 PyWebODM - gp1 - gp2 10k images -plancha
409 plancha - cap lahoussay 3 cams
408 session_2022_09_12_mayotte-fmr-surprise_Mavic2Pro_2
400 PyWebODM - uav_2023_07_20_terre-sainte_Mavic2Pro_02_2
399 PyWebODM - uav_2023_07_20_terre-sainte_Mavic2Pro_02_1
397 PyWebODM - uav_2023_06_26_bandamaji_Mavic2Pro_02_1
396 PyWebODM - uav_2023_07_01_Nioumadzaha-Mitsoudje_Mavic2Pro_01_1
392 PyWebODM - uav_2023_06_26_bandamaji_Mavic2Pro_01_2
391 PyWebODM - uav_2023_06_26_bandamaji_Mavic2Pro_01_1
390 PyWebODM - uav_2023_06_27_Mitsamiouli_Mavic2Pro_01_1
385 PyWebODM - uav_2023_06_27_Bangoua-Kouni_Mavic2Pro_01_1
378 PyWebODM - uav_2023_06_27_Vouvouni_Mavic2Pro_01_1
375 2023_06_29_Maloudja_drone_julien_Mission3
373 2023_06_29_Maloudja_drone_julien_Mission2
367 Maru Umbi GoPro
366 2023_05_01_Anakao_Prince_Anakao_drone_julien_Mission2
365 Demo Drone&Cap
364 Demo falaise
363 Massif des roses
362 corps_mort_ilot_bandrelé
361 corauxsp_ilot_bandrelé
360 Pchyseris_ilot_bandrelé
359 piscine_du_sud_LeptoriaPhrygia
358 Hélice_bateau
357 piscine_du_sud_Patch2
356 piscine-du-sud_ancre
355 piscine-du-sud_patch_corallien
354 Saint Paul Etang
353 Passe en S - corail col
352 salle_gonf_pole_opp
350 20230524_REU-ermitage_UAV-02_1
349 plancha ifaty
348 Ifaty - 07/05/2023
347 elements_pole_ope_PNMM
346 2023_05_06_Mangily_Hotel
345 2023_05_06_Mangily
344 2023_05_05_passe3
343 2023_05_05_GRT_passe2
342 2023_05_05
341 2023_05_07_massif_des_roses_gaetan_v2
340 2023_05_05_GRT_passe3
339 2023_05_05_passe_2
338 2023_05_05_Gauche_Manjaka_1
337 2023_05_07_massif_des_roses_gaetan
336 2023_05_05_Gauche_Manjaka_2
335 2023_05_05_Gauche_Manjaka
334 2023_05_03_sarodrano_IHSM_Belaza
333 2023_05_07_Mangily_cote2
332 2023_05_07_Mangily_cote1
331 2023_05_07_Mangily_Massif_Roses_drone_julien_vol2
330 2023_05_07_Mangily_Massif_Roses_drone_julien_vol1
329 Sarodrano_drone_matteo_sylvain
328 2023_05_04_Saint_Augustin_Pirogues
327 2023_05_04_GRT_micro_atoll_zone_B
326 Nosy_Velomitahy_2023_05_04
325 Nosy_Ve_2023_04_30_3
324 Nosy_Ve_2023_04_30_bis
323 Anakao_Prince_2020_04_29
322 Nosy_Ve_2
321 Nosy_Ve
320 demo_hab
318 pirogue_toliara
317 Poseidon_t4_TELEMAC
316 toliara_ihsm_droneSP_042023
315 arms_ihsm_jose
314 Mangrove IH.SM
313 DemoToliara
312 demo_geohab_artificialreef_ae_t1
311 demo_geohab_col_su_8
310 demo_geohab_col_su_14
309 demo_geohab_col_su_13
305 PyWebODM - uav_2022_10_24_aldabra-passe-dubois40m_Mavic2Pro_23
303 PyWebODM - 20221025_SYC-aldabra-arm01_UAV-02_32
302 PyWebODM - uav_2022_10_21_aldabra-arm01_Mavic2Pro_7
301 PyWebODM - uav_2022_09_20_aldabra-arm01_Mavic2Pro_1
300 PyWebODM - uav_2022_10_23_aldabra-arm06_Mavic2Pro_17
297 PyWebODM - uav_2022_10_24_aldabra-gigi_Mavic2Pro
296 PyWebODM - uav_2022_10_22_aldabra-passe-dubois_Mavic2Pro
295 PyWebODM - 20221022_SYC-aldabra-passe-dubois_UAV-02
275 dubois_test_cumul_modified_colors
274 dubois_test_cumul_no_modify_color
273 PyWebODM - dubois N4
271 photo_test_dubois_N4
192 2022_10_20_arm01_v1a_01
191 ARM01_Plancha_test
186 Test Ermitage Plancha
185 herbier total test
184 PorsmilinRTK_2022
181 Porsmilin_2015
180 Herbier 2023 suite
178 herbier
175 photogrametrie carnon
173 Mayotte_ib12
172 Mayotte_ib
170 Nosy_Komba
169 Sarodrano_2022_10_05
168 Anakao_2022_10_13
167 Ifaty_seagrass
166 test
162 PyWebODM - uav_2022_10_24_aldabra-passe-dubois40m_Mavic2Pro_24
161 PyWebODM - uav_2022_10_24_aldabra-passe-dubois40m_Mavic2Pro_23
158 PyWebODM - uav_2022_10_23_aldabra-arm06_Mavic2Pro_20
157 PyWebODM - uav_2022_10_23_aldabra-arm06_Mavic2Pro_19
140 PyWebODM - uav_2022_10_23_aldabra-arm06_Mavic2Pro_17
138 PyWebODM - uav_2022_10_22_aldabra-passe-dubois_Mavic2Pro_13
137 PyWebODM - uav_2022_10_22_aldabra-passe-dubois_Mavic2Pro_12
136 PyWebODM - 20221022_SYC-aldabra-passe-dubois_UAV-02_11
134 PyWebODM - uav_2022_10_22_aldabra-arm06_Mavic2Pro_15
133 PyWebODM - uav_2022_10_21_aldabra-arm01_Mavic2Pro_9
132 PyWebODM - uav_2022_10_21_aldabra-arm01_Mavic2Pro_8
131 PyWebODM - uav_2022_10_21_aldabra-arm01_Mavic2Pro_7
129 PyWebODM - uav_2022_09_20_aldabra-arm01_Mavic2Pro_6
127 PyWebODM - uav_2022_09_20_aldabra-arm01_Mavic2Pro_4
126 PyWebODM - uav_2022_09_20_aldabra-arm01_Mavic2Pro_3
125 PyWebODM - uav_2022_09_20_aldabra-arm01_Mavic2Pro_2
124 PyWebODM - uav_2022_09_20_aldabra-arm01_Mavic2Pro_1
123 PyWebODM - uav_2022_09_19_aldabra-arm01_Mavic2Pro_1
117 PyWebODM - session_2022_09_14_mayotte-fmr-ile-blanche_Mavic2Pro_2
116 PyWebODM - session_2022_09_14_mayotte-fmr-aeroport_Mavic2Pro_3
115 PyWebODM - session_2022_09_14_mayotte-fmr-aeroport_Mavic2Pro_1
114 PyWebODM - session_2022_09_13_mayotte-fmr-ngouja_Mavic2Pro_3
113 PyWebODM - session_2022_09_13_mayotte-fmr-ngouja_Mavic2Pro_2
112 PyWebODM - session_2022_09_13_mayotte-fmr-jollyroger_Mavic2Pro_1
111 PyWebODM - session_2022_09_12_mayotte-fmr-surprise_Mavic2Pro_3
110 PyWebODM - session_2022_09_12_mayotte-fmr-surprise_Mavic2Pro_2
109 PyWebODM - session_2022_09_12_mayotte-fmr-ile-blanche_Mavic2Pro_1
108 PyWebODM - session_2022_09_11_mayotte-fmr-musicalplage_Mavic2Pro_3
107 PyWebODM - session_2022_09_11_mayotte-fmr-jollyroger_Mavic2Pro_2
106 PyWebODM - session_2022_09_11_mayotte-fmr-jollyroger_Mavic2Pro_1
105 PyWebODM - session_2022_09_11_mayotte-fmr-aeroport_Mavic2Pro_2
104 PyWebODM - session_2022_09_11_mayotte-fmr-aeroport_Mavic2Pro_1
103 PyWebODM - session_2022_09_10_mayotte-fmr-musicalplage_Mavic2Pro_1v2
102 PyWebODM - session_2022_09_10_mayotte-fmr-musicalplage_Mavic2Pro_1
100 PyWebODM - session_2022_09_10_mayotte-fmr-jolly-roger_Mavic2Pro_3
99 PyWebODM - session_2022_09_10_mayotte-fmr-jollyroger_Mavic2Pro_2
53 Ifaty
51 semis_priorité_25052022
50 Mayotte - Ngouja 2
49 Semis_priorité_Grenoble_29072022
48 Mayotte - Ngouja
46 Galibier19072022
43 test_P_cilyndrica
41 2019_10_14_Morne
40 grandfond
37 Mayotte FMR - Ile Blanche
36 Mayotte FMR - Aéroport
31 08/07/2022 sortie test
16 Colonie coralienne 12
15 Colonie coralienne 3
14 Colonie coralienne 2
10 Ilôt Sancho
8 Bleaching Morne
60 58 20230524_REU-ermitage_UAV-02_4 ['a63e260a-6617-4220-a477-a31374b1cc8a']
Still logged-in!
a63e260a-6617-4220-a477-a31374b1cc8a
Downloading all.zip
Renaming all.zip to PROCESSED_DATA.zip
Saved PROCESSED_DATA.zip
Downloading report.pdf
Saved 000_photogrammetry_report.pdf
110 423 20230802_REU-ermitage_UAV-02_1 ['bc513f0c-73dc-4b85-906b-8f9700bc032a']
Still logged-in!
bc513f0c-73dc-4b85-906b-8f9700bc032a
Downloading all.zip
Renaming all.zip to PROCESSED_DATA.zip
Saved PROCESSED_DATA.zip
Downloading report.pdf
Saved 000_photogrammetry_report.pdf
111 422 20230705_REU-ermitage_UAV-02_1 ['56bddf6a-151f-4c46-9ec0-e3a57edcfe28']
Still logged-in!
56bddf6a-151f-4c46-9ec0-e3a57edcfe28
Downloading all.zip
Renaming all.zip to PROCESSED_DATA.zip
Saved PROCESSED_DATA.zip
Downloading report.pdf
Saved 000_photogrammetry_report.pdf
112 420 20230524_REU-ermitage_UAV-02_3 ['3698dca1-6279-4d7f-849d-1d8e7a63d2d7']
Still logged-in!
3698dca1-6279-4d7f-849d-1d8e7a63d2d7
Downloading all.zip
Renaming all.zip to PROCESSED_DATA.zip
Saved PROCESSED_DATA.zip
Downloading report.pdf
Saved 000_photogrammetry_report.pdf
113 419 20230524_REU-ermitage_UAV-02_2 ['03e7c014-9abc-4464-a186-ac64cd06ea3e']
Still logged-in!
03e7c014-9abc-4464-a186-ac64cd06ea3e
Downloading all.zip
Renaming all.zip to PROCESSED_DATA.zip
Saved PROCESSED_DATA.zip
Downloading report.pdf
Saved 000_photogrammetry_report.pdf
146 350 20230524_REU-ermitage_UAV-02_1 ['b42e5784-8cc6-42ef-9a2d-0ad754e54aac']
Still logged-in!
b42e5784-8cc6-42ef-9a2d-0ad754e54aac
Downloading all.zip
Renaming all.zip to PROCESSED_DATA.zip
Saved PROCESSED_DATA.zip
Downloading report.pdf
Saved 000_photogrammetry_report.pdf
[175]:
### Reformat dataframe
for entities in range(len(df)):
sess_path = os.path.join(
input, df.iloc[entities]["Identifier"].split("_\n")[0].split(":")[1]
)
df.iloc[entities, df.columns.get_loc("Title")] = df.iloc[entities][
"Title"
].replace(
"Images drone, UAV,",
"Orthophoto & DEM (MNE) issues d'images drone, UAV,",
)
df.iloc[entities, df.columns.get_loc("Description")] = df.iloc[entities][
"Description"
].replace(
"Jeu de données composé d’images",
"Ce jeu de données présente les résultats des traitements photogrammétriques d’images",
)
df.iloc[entities, df.columns.get_loc("Description")] = df.iloc[entities][
"Description"
].replace(
"Les vols ont été réalisés dans le but",
"Les traitements ont été réalisés avec le logiciel OpenDroneMap à partir des images brutes fournies dans la première version de ce DOI.\n<br />Les vols ont été réalisés dans le but",
)
df.iloc[entities, df.columns.get_loc("Description")] = df.iloc[entities][
"Description"
].replace(
"<br /><b>Le dépôt est composé",
"<br /><br />Le paramétrage du logiciel OpenDroneMap est partagé pour permettre la reproductibilité ou l'amélioration des traitements proposés:<br />\n[\n<br /> {\n<br /> 'name': 'orthophoto-resolution',\n<br /> 'value': 1\n<br /> },\n<br /> {\n<br /> 'name': 'auto-boundary',\n<br /> 'value': true\n<br /> },\n<br /> {\n<br /> 'name': 'dem-resolution',\n<br /> 'value': '2.0'\n<br /> },\n<br /> {\n<br /> 'name': 'dsm',\n<br /> 'value': true\n<br /> }\n<br /> ]\n<br />\n<br /><b>Le dépôt est composé",
)
df.iloc[entities, df.columns.get_loc("Description")] = df.iloc[entities][
"Description"
].replace(
"<br /> - METADATA.zip: Métadonnées au format ISO19115, Rapports avec miniatures des images de drone (dossier tb) et statistiques de vols.",
"<br /> - METADATA.zip: Métadonnées au format ISO19115, Rapports avec miniatures des images de drone (dossier tb) et statistiques de vols.\n<br /> - PROCESSED_DATA.zip: Orthophoto, DEM, nuages de points, ...",
)
df.iloc[entities, df.columns.get_loc("Description")] = df.iloc[entities][
"Description"
].replace(
"<br />│---------------- └─ tb",
"<br />│---------------- └─ tb\n<br />│-------- └─ PROCESSED_DATA",
)
df.iloc[entities, df.columns.get_loc("Description")] = df.iloc[entities][
"Description"
].replace(
"\n<br />Les dépôts suivants comprennent les données traitées avec OpenDroneMap",
"",
)
df.iloc[entities, df.columns.get_loc("Description")] = df.iloc[entities][
"Description"
].replace(
"\n<br />D'autres versions plus récentes de ce dépôt sont disponibles. Ces dernières comprennent notamment les données traitées avec OpenDroneMap",
"",
)
df.iloc[entities, df.columns.get_loc("Description")] = df.iloc[entities][
"Description"
].replace("RAW DATA", "PROCESSED DATA")
df.iloc[entities, df.columns.get_loc("Provenance")] = (
df.iloc[entities]["Provenance"] + "_\nprocess:Download from WebODM"
)
df.iloc[
entities, df.columns.get_loc("Format")
] = "resource:image/tiff_\ndistribution:image/tiff[GeoTIFF]"
df.iloc[entities, df.columns.get_loc("Data")] = (
"source:"
+ pdf_newname
+ "@"
+ os.path.join(sess_path, pdf_newname)
+ ","
+ file_newname
+ "@"
+ os.path.join(sess_path, file_newname)
+ "_\nsourceType:other"
)
[142]:
print(df.iloc[0]["Title"])
print(df.iloc[0]["Description"])
print(df.iloc[0]["Data"])
title:Orthophoto & DEM (MNE) issues d'images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230524 - 02_4
abstract:"Ce jeu de données présente les résultats des traitements photogrammétriques d’images de drone DJI Mavic 2 Pro UAV acquises sur le site de Ermitage, Saint-Gilles, Réunion à la date suivante : 20230524.
<br />Les traitements ont été réalisés avec le logiciel OpenDroneMap à partir des images brutes fournies dans la première version de ce DOI.
<br />Les vols ont été réalisés dans le but de créer des modèles numériques d'élévations pour cartographier la rugosité récifale du lagon jusqu'à la pente externe. Les données seront utilisées dans le cadre du projet TELEMAC.
<br />
<br /><br />Le paramétrage du logiciel OpenDroneMap est partagé pour permettre la reproductibilité ou l'amélioration des traitements proposés:<br />
[
<br /> {
<br /> 'name': 'orthophoto-resolution',
<br /> 'value': 1
<br /> },
<br /> {
<br /> 'name': 'auto-boundary',
<br /> 'value': true
<br /> },
<br /> {
<br /> 'name': 'dem-resolution',
<br /> 'value': '2.0'
<br /> },
<br /> {
<br /> 'name': 'dsm',
<br /> 'value': true
<br /> }
<br /> ]
<br />
<br /><b>Le dépôt est composé des éléments suivants:</b>
<br /> - 00_: Planche d'aperçu des images
<br /> - DCIM.zip: Images brutes issues du drone
<br /> - GPS.zip: Geopackage contenant l’emprise du survol ainsi que la géolocalisation des images accompagnées de leurs miniatures dans la table d’attribut en base64. RINEX et LLH issus d'un Emlid Reach M2 synchronisé avec la LED de navigation (événement envoyé dans le log du récepteur GNSS lors du déclenchement d'une image). Le but est de pouvoir réaliser un PPK (similaire au RTK en post-traitement) et ainsi disposer d'une position centrimétrique sur chaque image.
<br /> - METADATA.zip: Métadonnées au format ISO19115, Rapports avec miniatures des images de drone (dossier tb) et statistiques de vols.
<br /> - PROCESSED_DATA.zip: Orthophoto, DEM, nuages de points, ...
<br />
<br /><b>Arborescence d'origine:</b>
<br />│ └─ 20230524_REU-ermitage_UAV-02_4
<br />│-------- └─ DCIM
<br />│-------- └─ GPS
<br />│---------------- └─ base_2023_05_24_pascal
<br />│---------------- └─ reach_2023_05_24_drone
<br />│------------------------ └─ reachsylvai_raw_202305240249_RINEX_3_03
<br />│------------------------ └─ reachsylvai_raw_202305240330_RINEX_3_03
<br />│---------------- └─ reach_2023_05_24_rover
<br />│-------- └─ METADATA
<br />│---------------- └─ tb
<br />│-------- └─ PROCESSED_DATA
<br />
<br /><b>Informations de survol:</b>
<br />- Camera model and parameters:
<br /> Make: Hasselblad
<br /> Model: L1D-20c
<br /> Width: 5472
<br /> Height: 3648
<br /> Focal: 28
<br /> WhiteBalance: Manual
<br /> ExposureMode: Auto Exposure
<br /> ColoSpace: sRGB
<br /> EV: -0.7
<br /> MeteringMode: CenterWeightedAverage
<br /> Camera Pitch: -70.00
<br />
<br />- Survey informations:
<br /> No Images: 193
<br /> Median height: 70 meters
<br /> Survey area: 4.16 hectares
<br /> Survey from: 2023:05:24 09:53:20 to: 2023:05:24 10:03:57
<br />"_
purpose:" "_
info:"This study was funded by the European Regional Development Fund (ERDF) within the programme Interreg V 2014-2020 through the project G2OI"_
edition:"PROCESSED DATA"
source:000_photogrammetry_report.pdf@/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4/000_photogrammetry_report.pdf,PROCESSED_DATA.zip@/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4/PROCESSED_DATA.zip_
sourceType:other
[176]:
### Export processed data df to csv
# fname = out_prefix + '_zenodo-rawdata'
# df.to_csv(os.path.join(output, fname + ".csv"), quoting=1, quotechar='"', index=False)
df.to_csv(os.path.join(output, out_prefix + "_processed_data.csv"), quoting=1, quotechar='"', index=False)
[200]:
# Reload df from WebODM download as updated dataframe
df = {}
df = pd.read_csv(os.path.join(input, out_prefix + '_processed_data.csv'))
df.head()
[200]:
| Identifier | Title | Description | Subject | Creator | Date | Type | Language | SpatialCoverage | TemporalCoverage | Relation | Rights | Provenance | Format | Data | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | id:20230524_REU-ermitage_UAV-02_4_\ndoi:10.5281/zenodo.10037633 | title:Orthophoto & DEM (MNE) issues d'images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230524 -... | abstract:"Ce jeu de données présente les résultats des traitements photogrammétriques d’images de dro... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | NaN | 2023-05-24 09-53-20 - 2023-05-24 10-03-57 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/tiff_\ndistribution:image/tiff[GeoTIFF] | source:000_photogrammetry_report.pdf@/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4... |
| 1 | id:20230524_REU-ermitage_UAV-02_2_\ndoi:10.5281/zenodo.10037635 | title:Orthophoto & DEM (MNE) issues d'images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230524 -... | abstract:"Ce jeu de données présente les résultats des traitements photogrammétriques d’images de dro... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | NaN | 2023-05-24 09-05-25 - 2023-05-24 09-26-24 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/tiff_\ndistribution:image/tiff[GeoTIFF] | source:000_photogrammetry_report.pdf@/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2... |
| 2 | id:20230705_REU-ermitage_UAV-02_1_\ndoi:10.5281/zenodo.10037640 | title:Orthophoto & DEM (MNE) issues d'images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230705 -... | abstract:"Ce jeu de données présente les résultats des traitements photogrammétriques d’images de dro... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | NaN | 2023-07-05 10-33-06 - 2023-07-05 10-48-40 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/tiff_\ndistribution:image/tiff[GeoTIFF] | source:000_photogrammetry_report.pdf@/home/sylvain/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1... |
| 3 | id:20230524_REU-ermitage_UAV-02_3_\ndoi:10.5281/zenodo.10037645 | title:Orthophoto & DEM (MNE) issues d'images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230524 -... | abstract:"Ce jeu de données présente les résultats des traitements photogrammétriques d’images de dro... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | NaN | 2023-05-24 09-33-18 - 2023-05-24 09-49-53 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/tiff_\ndistribution:image/tiff[GeoTIFF] | source:000_photogrammetry_report.pdf@/home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_3... |
| 4 | id:20230802_REU-ermitage_UAV-02_1_\ndoi:10.5281/zenodo.10037649 | title:Orthophoto & DEM (MNE) issues d'images drone, UAV, Ermitage, Saint-Gilles, Réunion - 20230802 -... | abstract:"Ce jeu de données présente les résultats des traitements photogrammétriques d’images de dro... | theme[General]:TELEMAC,Réunion,Hermitage,Ermitage,Saint-Gilles,La Réunion,océan Indien,lagon,lagoon,c... | author:sylvain.poulain@ird.fr,pascal.mouquet@ird.fr,julien.barde@ird.fr_\npublisher:sylvain.poulain@i... | publication:2023-10-22_\nedition:2023-10-22 | dataset | fra | NaN | 2023-08-02 06-59-36 - 2023-08-02 07-16-47 | thumbnail:G2OI@https://raw.githubusercontent.com/IRDG2OI/.github/main/profile/logo_g2oi_final_1280px_... | useLimitation:Utilisation libre sous réserve de mentionner la source (a minima le nom du producteur) ... | statement:"- Camera model and parameters:\n Make: Hasselblad\n Model: L1D-20c\n Width: 5472\n Height:... | resource:image/tiff_\ndistribution:image/tiff[GeoTIFF] | source:000_photogrammetry_report.pdf@/home/sylvain/HDD5To/DATA/TELEMAC/20230802_REU-ermitage_UAV-02_1... |
[177]:
data_zip = []
sources = df.iloc[0]['Data'].split('_\n')[0].split(':')[1].split(',')
for data_file in sources:
data_zip.append(data_file.split('@')[0])
print(data_zip)
['000_photogrammetry_report.pdf', 'PROCESSED_DATA.zip']
[185]:
### UNZIP PROCESSED_DATA
from zip_helper import *
for entities in range(len(df)):
print("unzip PROCESSED_DATA.zip in: ", sess_path)
sess_path = os.path.join(
input, df.iloc[entities]["Identifier"].split("_\n")[0].split(":")[1]
)
try:
unzip(sess_path, 'PROCESSED_DATA.zip', 'PROCESSED_DATA/PHOTOGRAMMETRY/')
except:
print("no_zip in: ", sess_path)
print("Remove zip")
try:
os.remove(os.path.join(sess_path, 'PROCESSED_DATA.zip'))
except:
print("no_zip to remove in: ", sess_path)
print("Recreate zip")
zip_folder(os.path.join(sess_path, 'PROCESSED_DATA'), os.path.join(sess_path, 'PROCESSED_DATA'))
unzip PROCESSED_DATA.zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4
no_zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4
Remove zip
no_zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4
Recreate zip
unzip PROCESSED_DATA.zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_4
no_zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2
Remove zip
no_zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2
Recreate zip
unzip PROCESSED_DATA.zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_2
no_zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1
Remove zip
no_zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1
Recreate zip
unzip PROCESSED_DATA.zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230705_REU-ermitage_UAV-02_1
Remove zip
Recreate zip
unzip PROCESSED_DATA.zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230524_REU-ermitage_UAV-02_3
Remove zip
Recreate zip
unzip PROCESSED_DATA.zip in: /home/sylvain/HDD5To/DATA/TELEMAC/20230802_REU-ermitage_UAV-02_1
Remove zip
Recreate zip
[178]:
df.iloc[0]['Identifier'].split('\n')[1].split('.')[-1]
[178]:
'10037633'
Geoflow to upload processed data
[143]:
# ### Execute geoflow :: upload to geoflow
# # config_raw = os.path.join(output, out_prefix + '_rawfiles-config.json')
# pdata = os.path.join(output, 'comores_processed.json')
# # run_geoflow(config_raw, geoflow_folder)
# os.chdir(geoflow_folder)
# # subprocess.run(["R", "-e", "geoflow::initWorkflow('{}')".format(config_raw)], capture_output=False)
# subprocess.call(["R", "-e", "geoflow::executeWorkflow('{}')".format(pdata)],
# stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT,)
# os.chdir(dronemdt_folder)
[198]:
### Upload processed_data to Zenodo
print("#### Upload zip files to Zenodo")
base_url = "https://zenodo.org/api/"
for zipul in range(len(df)):
print(" Check", df.iloc[zipul]['Identifier'].split('\n')[0])
zenodo_baseurl = base_url
data_zip = []
sources = df.iloc[zipul]['Data'].split('_\n')[0].split(':')[1].split(',')
for data_file in sources:
data_zip.append(data_file.split('@')[0])
print(data_zip)
print("New deposit version")
### New version
doi_raw = df.iloc[zipul]['Identifier'].split('\n')[1].split('.')[-1]
newv = zen_newversion(zenodo_baseurl, ACCESS_TOKEN, str(doi_raw))
# r = check_token(zenodo_baseurl, ACCESS_TOKEN)
zenval = zenvar(newv)
print("New doi version:"+zenval[1])
print("Write DOI to dataframe")
if 'id:' in df.iloc[zipul]['Identifier']:
pass
else:
df.iloc[zipul, df.columns.get_loc('Identifier')] = "id:" + df.iloc[zipul]['Identifier'].split('\n')[0].split('id:')[1] + "_\ndoi:" + zenval[1]
df.iloc[zipul, df.columns.get_loc("Provenance")] = df.iloc[zipul]["Provenance"] + "_\nprocess:Raw dataset uploaded to " + base_url.split('api')[0] + "record/" + str(zenval[2])
print("Enrich upload with metadata")
zen_metadata = zenmdt(zenodo_baseurl, ACCESS_TOKEN, zenval[2], df, zipul)
# zen_metadata.text
print("upload data")
print(" Trying upload number: 1")
for file in data_zip:
file_list = [file]
if file in newv.json()["files"]:
print("#### file {} already in repository.\nSKIP upload!".format(file))
pass
else:
print("#### file {} not in repository.\n#### upload!".format(file))
zen_upload = zenul(zenval[0], ACCESS_TOKEN, os.path.join(input, df.iloc[zipul]['Identifier'].split('_\n')[0].split(':')[1]), file_list)
# zen_upload = zenul(zenvar(newv), ACCESS_TOKEN, os.getcwd(), test_list)
# zen_upload = zenul2(zenodo_baseurl, str(zenval[-1]), ACCESS_TOKEN, os.path.join(input, df.iloc[zipul]['Identifier'].split('_\n')[0].split(':')[1]), data_zip)
print(" Upload status OK? " + str(zen_upload.ok))
print(" Control files: first pass")
ul_count = 1
while zen_upload.status_code != 201:
if zen_upload.status_code == 404:
print(" Version doesn't exists ! Please check your record_id")
break
else:
ul_count += 1
print(" Retry number: " + str(ul_count))
zen_upload = zenul(zenval[0], ACCESS_TOKEN, os.path.join(input, df.iloc[zipul]['Identifier'].split('_\n')[0].split(':')[1]), file_list)
if zen_upload.status_code == 403:
print(" permission denied!")
time.sleep(10)
### Too agressive
# print("Control files: second pass")
# if zen_upload.status_code == 201:
# # files_ul = zenfiles(zenodo_baseurl, ACCESS_TOKEN, str(zenval[2]))
# while len(zenfiles(zenodo_baseurl, ACCESS_TOKEN, str(zenval[2])).json()) != len(data_zip):
# if zen_upload.status_code == 404:
# print("Version doesn't exists ! Please check your record_id")
# break
# else:
# ul_count += 1
# print("Retry number: " + str(ul_count))
# zen_upload = zenul(zenval[0], ACCESS_TOKEN, os.path.join(input, df.iloc[zipul]['Identifier'].split('_\n')[0].split(':')[1]), data_zip)
# # files_ul = zenfiles(zenodo_baseurl, ACCESS_TOKEN, str(zenval[2]))
### Publish to Zenodo
# print("Publish record")
# zenpublish(zenodo_baseurl, ACCESS_TOKEN, zenval[2])
#### Upload zip files to Zenodo
['000_photogrammetry_report.pdf', 'PROCESSED_DATA.zip']
New deposit version
New doi version:10.5281/zenodo.10039105
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: 000_photogrammetry_report.pdf
Upload status OK? True
Control files: first pass
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
Upload status OK? True
Control files: first pass
['000_photogrammetry_report.pdf', 'PROCESSED_DATA.zip']
New deposit version
New doi version:10.5281/zenodo.10039106
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: 000_photogrammetry_report.pdf
Upload status OK? True
Control files: first pass
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
permission denied!
Retry number: 3
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
permission denied!
Retry number: 4
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
permission denied!
Retry number: 5
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
permission denied!
Retry number: 6
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
permission denied!
Retry number: 7
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
['000_photogrammetry_report.pdf', 'PROCESSED_DATA.zip']
New deposit version
New doi version:10.5281/zenodo.10039108
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: 000_photogrammetry_report.pdf
Upload status OK? True
Control files: first pass
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
permission denied!
Retry number: 3
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
['000_photogrammetry_report.pdf', 'PROCESSED_DATA.zip']
New deposit version
New doi version:10.5281/zenodo.10039111
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: 000_photogrammetry_report.pdf
Upload status OK? True
Control files: first pass
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
Upload status OK? True
Control files: first pass
['000_photogrammetry_report.pdf', 'PROCESSED_DATA.zip']
New deposit version
New doi version:10.5281/zenodo.10039112
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: 000_photogrammetry_report.pdf
Upload status OK? True
Control files: first pass
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
Upload status OK? True
Control files: first pass
['000_photogrammetry_report.pdf', 'PROCESSED_DATA.zip']
New deposit version
New doi version:10.5281/zenodo.10039113
Write DOI to dataframe
Enrich upload with metadata
upload data
Trying upload number: 1
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: 000_photogrammetry_report.pdf
Upload status OK? True
Control files: first pass
#### file {} not in repository.
#### upload!
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
Upload status OK? False
Control files: first pass
Retry number: 2
Sleep 1 second before new upload
upload: PROCESSED_DATA.zip
[199]:
### Export processed data df to csv
# fname = out_prefix + '_zenodo-rawdata'
# df.to_csv(os.path.join(output, fname + ".csv"), quoting=1, quotechar='"', index=False)
df.to_csv(os.path.join(output, out_prefix + "_processed_data_updated_doi.csv"), quoting=1, quotechar='"', index=False)
# # config_raw = os.path.join(output, out_prefix + '_rawfiles-config.json')
[214]:
# Reload df from WebODM download as updated dataframe
df = {}
df = pd.read_csv(os.path.join(input, out_prefix + '_processed_data_updated_doi.csv'))
# df.head()
### Check DOI's
for doi in range(len(df)):
print(df.iloc[doi]["Identifier"])
id:20230524_REU-ermitage_UAV-02_4_
doi:10.5281/zenodo.10038383
id:20230524_REU-ermitage_UAV-02_2_
doi:10.5281/zenodo.10038464
id:20230705_REU-ermitage_UAV-02_1_
doi:10.5281/zenodo.10039108
id:20230524_REU-ermitage_UAV-02_3_
doi:10.5281/zenodo.10039111
id:20230802_REU-ermitage_UAV-02_1_
doi:10.5281/zenodo.10039112
id:20230524_REU-ermitage_UAV-02_1_
doi:10.5281/zenodo.10039113
[211]:
df.iloc[0]["Description"].split("edition:")[1].split("_\n")[0].replace('"', '')
[211]:
'PROCESSED DATA'
[ ]: