How does your MIDI target's bulk dump work?

Question

We're dreaming of a new feature for MD which will allow users to parse bulk-dump sysex and send the values to individual controls in an MD layout. We imagine that if you can specify these for each control it's enough:

1. byte offset: where the control's value starts in the sysex dump
2. value length: how many bytes the value occupies in the sysex dump
3. LSB-first: if the bytes are reversed

For those of you know know the MIDI spec for your target well, would these values be enough per control? If not, how would we parse your MIDI target's sysex bulk dump?

Answer 1

MIDI Target Manufacturer(s): Clavia/Nord

For the NORD lead 2x Sysex dump for a patch (using DUMP ONE hard button):

I Have highlighted where the Control bytes start and highlighted alternate pairs, they are LSB first.  each control is 2 bytes and have attached the list of controls from the manual

Sysex: F0 33 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0B 02 01 05 0E 03 00 01 00 00 01 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 F7

This is better than the All CONTROLLERS returned message which is not so easily understood - even questions to NORD have not been very helpful about what the bytes mean.

Images

Screen Shot 2016-09-02 at 2.23.02 PM.png

Answer 2

MIDI Target Manufacturer(s): Clavia/Nord

For the NORD lead 2x Sysex dump for a patch (using DUMP ONE hard button):

I Have highlighted where the Control bytes start and highlighted alternate pairs, they are LSB first.  each control is 2 bytes and have attached the list of controls from the manual

Sysex: F0 33 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0B 02 01 05 0E 03 00 01 00 00 01 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 F7

This is better than the All CONTROLLERS returned message which is not so easily understood - even questions to NORD have not been very helpful about what the bytes mean.

Images

Screen Shot 2016-09-02 at 2.23.02 PM.png

Answer 3

Roland Boutique JP-08, JU-06 and JX-03

I am not very expert in the sysex language so I answer with a question: would your proposed method work with Roland Boutique synths' sysex?

The sysex bulk dump in Roland Boutique JU-06 is split in different messages by function (that I have indicated in square brackets); for example the bulk dump of a single preset is:

[LFO]    
F0 41 10 00 00 00 1D 12 03 00 06 00 01 0E 03 0B 00 00 00 00 00 00 00 00 00 00 5A F7
 

[DCO]

F0 41 10 00 00 00 1D 12 03 00 07 00 00 00 00 00 0F 0D 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 59 F7
 

[VCF]

F0 41 10 00 00 00 1D 12 03 00 08 00 02 09 06 06 01 01 00 01 06 04 00 00 0C 02 00 00 00 00 00 00 00 00 00 00 43 F7
 

[VCA]

F0 41 10 00 00 00 1D 12 03 00 09 00 00 01 0F 0F 00 00 00 00 00 00 00 00 00 00 55 F7
 

[ENV]

F0 41 10 00 00 00 1D 12 03 00 0A 00 02 00 06 0E 07 07 03 0C 00 00 00 00 00 00 00 00 00 00 40 F7
 

[Effects]

F0 41 10 00 00 00 1D 12 03 00 10 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 6C F7
 

[Controller]

F0 41 10 00 00 00 1D 12 03 00 11 00 00 00 06 04 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 60 F7 

For more reference please see: http://inaptlycrafted.blogspot.com/2015/12/roland-boutique-ju-08-midi-system.html

Thanks

Giovanni

Comments

Thanks man, that's exactly what I've been expecting: that this stuff is NOT regular. I'll study this further, much appreciated Giovanni!
Dan

---

Absolutely not regular, but this is! You're welcome and thanks for the efforts

Answer 4

Roland Boutique JP-08, JU-06 and JX-03

I am not very expert in the sysex language so I answer with a question: would your proposed method work with Roland Boutique synths' sysex?

The sysex bulk dump in Roland Boutique JU-06 is split in different messages by function (that I have indicated in square brackets); for example the bulk dump of a single preset is:

[LFO]    
F0 41 10 00 00 00 1D 12 03 00 06 00 01 0E 03 0B 00 00 00 00 00 00 00 00 00 00 5A F7
 

[DCO]

F0 41 10 00 00 00 1D 12 03 00 07 00 00 00 00 00 0F 0D 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 59 F7
 

[VCF]

F0 41 10 00 00 00 1D 12 03 00 08 00 02 09 06 06 01 01 00 01 06 04 00 00 0C 02 00 00 00 00 00 00 00 00 00 00 43 F7
 

[VCA]

F0 41 10 00 00 00 1D 12 03 00 09 00 00 01 0F 0F 00 00 00 00 00 00 00 00 00 00 55 F7
 

[ENV]

F0 41 10 00 00 00 1D 12 03 00 0A 00 02 00 06 0E 07 07 03 0C 00 00 00 00 00 00 00 00 00 00 40 F7
 

[Effects]

F0 41 10 00 00 00 1D 12 03 00 10 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 6C F7
 

[Controller]

F0 41 10 00 00 00 1D 12 03 00 11 00 00 00 06 04 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 60 F7 

For more reference please see: http://inaptlycrafted.blogspot.com/2015/12/roland-boutique-ju-08-midi-system.html

Thanks

Giovanni

Answer 5

java script parsing prototype

I’ve done some sysex parsing. You cannot just assume a byte position within the sysex dump. I’ve seen sysex messages where messages have different length thus a need to branch (have a logic) the parsing. If we were able to providee JSON/js file to Dan with a definition of a sysex preset dump (ONLY Single preset) 

Here is how it could work (MDP Midi designer pro) 

1. Make sure that js can be called from within APP if not then JSON is other option (like this http://will.thimbleby.net/script/)

2. Establish function prototypes (interfaces)  that would ensure that MDP sends and gets EXPECTED structures: Example 

function parsesysex (msg string,layoutcontrols)
sysex.log (‘send debugging message to a regular midi log')
// perse message here loops etc long execution will be punished
for each control in layoltcontrols   
  output.add(control.name,value)
set success so in the case of interruption MDP knows whether the message was parsed in full or partially
throw exception {
 if error throw exception (parsing error, short message etc) 
}  
return output
} 

3. MDP then parses the key/value function to the each control on layout 

4. MDP throws an error to the log if there was problem with parsing 

5. MDP interrupts the execution should the execution take more than 2-3 seconds 

6. MDP can provide success message to log to say: “successfully set 34 values" 

7. The source would be set in the type of the control
7b. this subControl could be executed from the Patch changer 
The json/js file could be zipped so that it can be emailed as templates are sent.   

8. MDP would know where to put this file    

Let me know what do you think 

Comments

This is an impressive message and thought. I'll definitely think about it further. It might be outside what's allowed in the App Store (have to check on that) and it's definitely completely antithetical to the "no scripting" approach that I've achieved thus far. The no-scripting approach protects our users and also protects me from going way over my head into creating a coding platform. That said, I'd probably go the OTHER way: to provide programmers with a VERY simple API + Unit Tests so they can do the conversion from Sysex to individual MIDI messages using Swift. These extensions would then get included inside the app itself so regular users could use them. While I do love Javascript, it sounds like we're firmly on the terrain of PROGRAMMING, and if that's the case, Swift (or Objective-C) would be the way to go. Or one way to go. Thoughts? Thanks again for this extensive answer and suggestion, bladuck!
Dan

---

Sorry, that should've read "Swift or Objective-C or C, really"... Thanks

---

An oldish thread, but I'm interested that with the inclusion of Streambyter in MD, did any of you succesfully use this to parse your patch dump Sysex? I've had good results when the controls are using CCs, but NRPNs have proven to be tricky. Because NRPN essentially use the same 3 or 4 CC#s (98/99/06(/32)), they require SB's delay flag to separate messages. However while this method works in MidiFire, it seems the Delays are not fed into MD via the input section. Has anyone found a successful workaround for NRPN delivery from sysex? Cheers in advance.

---

Yeah, we don't handle delays in SB for incoming MIDI, yet. So you need delays to be able to address different NRPNs inside MD?

Answer 6

java script parsing prototype

I’ve done some sysex parsing. You cannot just assume a byte position within the sysex dump. I’ve seen sysex messages where messages have different length thus a need to branch (have a logic) the parsing. If we were able to providee JSON/js file to Dan with a definition of a sysex preset dump (ONLY Single preset) 

Here is how it could work (MDP Midi designer pro) 

1. Make sure that js can be called from within APP if not then JSON is other option (like this http://will.thimbleby.net/script/)

2. Establish function prototypes (interfaces)  that would ensure that MDP sends and gets EXPECTED structures: Example 

function parsesysex (msg string,layoutcontrols)
sysex.log (‘send debugging message to a regular midi log')
// perse message here loops etc long execution will be punished
for each control in layoltcontrols   
  output.add(control.name,value)
set success so in the case of interruption MDP knows whether the message was parsed in full or partially
throw exception {
 if error throw exception (parsing error, short message etc) 
}  
return output
} 

3. MDP then parses the key/value function to the each control on layout 

4. MDP throws an error to the log if there was problem with parsing 

5. MDP interrupts the execution should the execution take more than 2-3 seconds 

6. MDP can provide success message to log to say: “successfully set 34 values" 

7. The source would be set in the type of the control
7b. this subControl could be executed from the Patch changer 
The json/js file could be zipped so that it can be emailed as templates are sent.   

8. MDP would know where to put this file    

Let me know what do you think