haiku/docs/develop/midi/testing.html

567 lines
26 KiB
HTML

<HTML>
<BODY>
<H1>Testing the Midi Kit</H1>
<P>Most of the OpenBeOS source code has unit tests in the current/src/tests
directory. I looked into building CppUnit tests for the midi2 kit, but decided
that it doesn't really make much sense. Unit tests work best if you can test
something in isolation, but in the case of the midi2 kit this is very hard to
achieve. Because the classes from libmidi2.so always need to talk to the
midi_server, the tests depend on too many external factors. The available
endpoints, for example, will differ from system to system. The spray and hook
functions are difficult to test this way, too.</P>
<P>So instead of a CppUnit test suite, here is a list of manual tests that I
performed when developing the midi2 kit:</P>
<HR SIZE="1">
<H2>Registering the application</H2>
<P><I>Required:</I> Client app that calls BMidiRoster::MidiRoster()</P>
<UL>
<LI><P>When a client app starts, it should first receive mNEW notifications for
all endpoints in the system (even unregistered remotes), followed by mCON
notifications for all connections in the system (even those between two
unregistered local endpoints from another app).</P></LI>
<LI><P>Send invalid Mapp message (without messenger). The midi_server ignores
the request, and the client app blocks forever.</P></LI>
<LI><P>Fake a delivery error for the mNEW notifications and the mAPP reply.
(Add a snooze() in the midi_server's OnRegisterApplication(). While it is
snoozing, Ctrl-C the client app. Now the server can't deliver the message and
will unregister the application again.)</P></LI>
<LI><P>Kill the server. Start the client app. It should realize that the server
is not running, and return from MidiRoster(); it does not block
forever.</P></LI>
<LI><P>Note: The server does not protect against sending two or more Mapp
messages; it will add a new app_t object to the roster and it will also send
out the mNEW and mCON notifications again.</P></LI>
<LI><P>Verify that when the client app quits, the BMidiRoster instance is
destroyed by the BMidiRosterKiller. The BMidiRosterLooper is also destroyed,
along with any endpoint objects from its list. We don't destroy endpoints with
a refcount > 0, but print a warning message on stderr instead.</P></LI>
<LI><P>When the app quits before it has created a BMidiRoster instance, the
BMidiRosterKiller should do nothing.</P></LI>
</UL>
<HR SIZE="1">
<H2>Creating endpoints</H2>
<P><I>Required:</I> Client app that creates a new BMidiLocalProducer and/or
BMidiLocalConsumer</P>
<UL>
<LI><P>Send invalid Mnew message (missing fields). The server will return an
error code.</P></LI>
<LI><P>Don't send reply from midi_server. The client receives a B_NO_REPLY
error.</P></LI>
<LI><P>If something goes wrong creating a new local endpoint, you still get a
new BMidiEndpoint object (but it is not added to BMidiRosterLooper's internal
list of endpoints). Verify that its ID() function returns 0, and IsValid()
returns false. Verify that you can Release() it without crashing into the
debugger (i.e. the reference count of the new object should be 1).</P></LI>
<LI><P>Snooze in midi_server's OnCreateEndpoint() before sending reply to
client to simulate heavy processor load. Client should timeout. When done
snoozing, server fails to deliver the reply because the client is no longer
listening, and it unregisters the app.</P></LI>
<LI><P>Note: if you kill the client app with Ctrl-C before the server has sent
its reply, SendReply() still returns okay, and the midi_server adds the
endpoint, even though the corresponding app is dead. There is not much we can
do to prevent that (but it is not really a big deal).</P></LI>
<LI><P>Start the test app from two different Terminals. Verify that the new
local endpoint of app1 is added to the BMidiRosterLooper's list of endpoints,
and that its "isLocal" flag is true. Verify that when you start the second app,
it immediately receives mNEW notifications for the first app's endpoints. It
should also create BMidiEndpoint proxy objects for these endpoints with
"isLocal" set to false, and add them its own list. Vice versa for the endpoints
that app2 creates. Verify that the "registered" field in the mNEW notification
is false, because newly created endpoints are not registered yet. The
"properties" field should contain an empty message.</P></LI>
<LI><P>Start server. Start client app. The app makes new endpoints and the
server adds them to the roster. Ctrl-C the app. Start client app again. The new
client first receives mNEW notifications for the old app's endpoints. When the
new app tries to create its own endpoints, the server realizes that the old app
is dead, and sends mDEL notifications for the now-defunct endpoints.</P></LI>
<LI><P>The test app should now create 2 endpoints. Let the midi_server snooze
during the second create message, so the app times out. The server now
unregisters the app and purges its first endpoint (which was successfully
created).</P></LI>
<LI><P>The test app should now create 3 endpoints. Let the midi_server snooze
during the second create message, so the app times out. (It also times out when
sending the create request for the 3rd endpoint, because the server is still
snoozing.) Because it cannot send a reply for the 2nd create message, the
server now unregisters the app and purges its first endpoint (which was
successfully created). Then it processes the create request for the 3rd
endpoint, but ignores it because the app is now no longer registered with the
server.</P></LI>
<LI><P>Purging endpoints. The test app should now create 2 endpoints. Let the
midi_server snooze during the _fourth_ create message. Run the server. Run the
test app. Run the test app again in a second Terminal. The server times out,
and unregisters the second app. The first app should receive an mDEL
notification. Repeat, but now the test app should make 3 endpoints and the
server fails on the _sixth_ endpoint. The first app now receives 2 mDEL
notifications.</P></LI>
<LI><P>You should be allowed to pass NULL into the BMidiLocalProducer and
BMidiLocalConsumer constructor.</P></LI>
<LI><P>Let the midi_server assign random IDs to new endpoints; the
BMidiRosterLooper should sort the endpoints by their IDs when it adds them to
its internal list.</P></LI>
</UL>
<HR SIZE="1">
<H2>Deleting endpoints</H2>
<P><I>Required:</I> client app that creates one or more endpoints and
Release()'s them</P>
<UL>
<LI><P>Verify that Acquire() increments the endpoint's refcount and Release()
decrements it. When you Release() a local endpoint so its refcount becomes
zero, the client sends an Mdel request to the server. When you Release() a
local endpoint too many times, your app jumps into the debugger.</P></LI>
<LI><P>Send an Mdel request with an invalid ID to the server. Examples of
invalid IDs: -1, 0, 1000 (or any other large number).</P></LI>
<LI><P>Start the test app from two different Terminals. Note that when one of
the apps Release()'s its endpoints, the other receives corresponding mDEL
notifications.</P></LI>
<LI><P>Snooze in midi_server's OnCreateEndpoint() before sending reply to
"create endpoint" request. The client will timeout and the server will
unregister the app. Now have the client Release() the endpoint. This sends a
"delete endpoint" request to the server, which ignores the request because the
app is no longer registered.</P></LI>
<LI><P>Override BMidiLocalProducer and BMidiLocalConsumer, and provide a public
destructor. Call "delete prod; delete cons;" from your code, instead of using
Release(). Your app should drop into the debugger.</P></LI>
<LI><P>Start the client app and let it make its endpoints. Kill the server.
Release() the endpoints. The server doesn't run, so the Mdel request never
arrives, but the BMidiEndpoint objects should be deleted regardless.</P></LI>
<LI><P>Start the test app from two different Terminals, and let them make their
endpoints. Quit the apps (using the Deskbar's "Quit Application" menu item).
Verify that both clean up and exit correctly. App1 removes its own endpoint
from the BMidiRosterLooper's list of endpoints and sends an 'mDEL' message to
the server, which passes it on to app2. In response, app2 removes the proxy
object from its own list and deletes it. Again, vice versa for the endpoint
from app2.</P></LI>
<LI><P>Start both apps again and wait until they have notified each other about
the endpoints. Ctrl-C app1, and restart it. Verify that app1 receives the
'mNEW' messages and creates proxies for these remote endpoints. Both apps
should receive an 'mDEL' message for app1's old endpoint (because the
midi_server realizes it no longer exists and purges it), and remove it from
their lists accordingly.</P></LI>
</UL>
<HR SIZE="1">
<H2>Changing attributes</H2>
<P><I>Required:</I> Client app that creates an endpoint and calls Register(),
Unregister(), SetName(), and SetLatency()</P>
<UL>
<LI><P>Send an Mchg request with an invalid ID to the server.</P></LI>
<LI><P>Register() a local endpoint that is already registered. This does not
send a message to the server and always returns B_OK. Likewise for
Unregister()ing a local endpoint that is not registered.</P></LI>
<LI><P>Register() or Unregister() a remote endpoint, or an invalid local
endpoint. That should immediately return an error code.</P></LI>
<LI><P>Verify that BMidiRoster::Register() does the same thing as
BMidiEndpoint::Register(). Also for BMidiRoster::Unregister() and
BMidiEndpoint::Unregister().</P></LI>
<LI><P>If you pass NULL into BMidiRoster::Register() or Unregister(), the
functions immediately return with an error code.</P></LI>
<LI><P>SetName() should ignore NULL names. When you call it on a remote
endpoint, SetName() should do nothing. SetName() does not send a message if the
new name is the same as the current name.</P></LI>
<LI><P>SetLatency() should ignore negative values. SetLatency() does not send a
message if the new latency is the same as the current latency. (Since
SetLatency() lives in BMidiLocalConsumer, you can never use it on remote
endpoints.)</P></LI>
<LI><P>Kill the server after making the new endpoint, and call Register(). The
client app should return an error code. Also for Unregister(), SetName(),
SetLatency(), and SetProperties().</P></LI>
<LI><P>Snooze in the midi_server's OnChangeEndpoint() before sending the reply
to the client. Both sides will flag an error. No mCHG notifications will be
sent. The server unregisters the app and purges its endpoints.</P></LI>
<LI><P>Verify that other apps will receive mCHG notifications when the test app
successfully calls Register(), Unregister(), SetName(), and SetLatency(), and
that they modify the corresponding BMidiEndpoint objects accordingly. Since
clients are never notified when they change their own endpoints, they should
ignore the notifications that concern local endpoints. Latency changes should
be ignored if the endpoint is not a consumer.</P></LI>
<LI><P>Send an Mchg request with only the "midi:id" field, so no "midi:name",
"midi:registered", "midi:latency", or "midi:properties". The server will still
notify the other apps, although they will obviously ignore the notification,
because it doesn't contain any useful data.</P></LI>
<LI><P>The Mchg request is overloaded to change several attributes. Verify that
changing one of these attributes, such as the latency, does not overwrite/wipe
out the others.</P></LI>
<LI><P>Start app1. Wait until it has created and registered its endpoint. Start
app2. During the initial handshake, app2 should receive an 'mNEW' message for
app1's endpoint. Verify that the "refistered" field in this message is already
true, and that this is passed on correctly to the new BMidiEndpoint proxy
object.</P></LI>
<LI><P>GetProperties() should return NULL if the message parameter is
NULL.</P></LI>
<LI><P>The properties of new endpoints are empty. Create a new endpoint and
call GetProperties(). The BMessage that you receive should contain no
fields.</P></LI>
<LI><P>SetProperties() should return NULL if the message parameter is NULL. It
should return an error code if the endpoint is remote or invalid. It should
work fine on local endpoints, registered or not. SetProperties() does not
compare the contents of the new BMessage to the old, so it will always send out
the change request.</P></LI>
<LI><P>If you Unregister() an endpoint that is connected, the connection should
not be broken.</P></LI>
</UL>
<HR SIZE="1">
<H2>Consulting the roster</H2>
<P><I>Required:</I> Client app that creates several endpoints, and registers
some of them (not all), and uses the BMidiRoster::FindEndpoint() etc functions
to examine the roster.</P>
<UL>
<LI><P>Verify that FindEndpoint() returns NULL if you pass it:</P>
<UL>
<LI>invalid ID (localOnly = false)</LI>
<LI>invalid ID (localOnly = true)</LI>
<LI>remote non-registered endpoint (localOnly = false)</LI>
<LI>remote non-registered endpoint (localOnly = true)</LI>
<LI>remote registered endpoint (localOnly = true)</LI>
</UL><BR>
<P>Verify that FindEndpoint() returns a valid BMidiEndpoint object if you pass
it:</P>
<UL>
<LI>local non-registered endpoint (localOnly = false)</LI>
<LI>local non-registered endpoint (localOnly = true)</LI>
<LI>local registered endpoint (localOnly = false)</LI>
<LI>local registered endpoint (localOnly = true)</LI>
<LI>remote registered endpoint (localOnly = false)</LI>
</UL><BR>
</LI>
<LI><P>Verify that FindConsumer() works just like FindEndpoint(), but that it
also returns NULL if the endpoint with the specified ID is not a consumer.
Likewise for FindProducer().</P></LI>
<LI><P>Verify that NextEndpoint() returns NULL if you pass it NULL. It also
returns NULL if no more endpoints exist. Otherwise, it returns a BMidiEndpoint
object, bumps the endpoint's reference count, and sets the "id" parameter to
the ID of the endpoint. NextEndpoint() should never return local endpoints
(registered or not), nor unregistered remote endpoints. Verify that negative
"id" values also work.</P></LI>
<LI><P>Verify that you can safely call the Find and Next functions without
having somehow initialized the BMidiRoster first (by making a new endpoint, for
example). The functions themselves should call MidiRoster() and do the
handshake with the server.</P></LI>
<LI><P>The Find and Next functions should bump the reference count of the
BMidiEndpoint object that they return. However, they should not (inadvertently)
modify the refcounts of any other endpoint objects.</P></LI>
<LI><P>Get a BMidiEndpoint proxy for a remote published endpoint. Release().
Now it should not be removed from the endpoint list or even be deleted, even
though its reference count dropped to zero.</P></LI>
<LI><P>Start app1. Start app2. App2 gets a BMidiEndpoint proxy for a remote
endpoint from app1. Ctrl-C app1. Start app1 again. Now app2 receives an mDEL
message for app1's old endpoint. Verify that the endpoint is removed from the
endpoint list, but not deleted because its reference count isn't zero. If app2
now Release()s the endpoint, the BMidiEndpoint object should be deleted. Try
again, but now Release() the endpoint before you Ctrl-C; now it should be
deleted and removed from the list when you start app1 again.</P></LI>
</UL>
<HR SIZE="1">
<H2>Making/breaking connections</H2>
<P><I>Required:</I> Client app that creates a producer and consumer endpoint,
optionally registers them, consults the roster for remote endpoints, and makes
various kinds of connections.</P>
<UL>
<LI><P>Test the following for BMidiProducer::Connect():</P>
<UL>
<LI>Connect(NULL)</LI>
<LI>Connect(invalid consumer)</LI>
<LI>Connect() using an invalid producer</LI>
<LI>Send Mcon request with invalid IDs</LI>
<LI>Kill the midi_server just before you Connect()</LI>
<LI>Let the midi_server snooze, so the connect request times out</LI>
<LI>Have the midi_server return an error result code</LI>
<LI>On successful connect, verify that the consumer is added to the producer's
list of endpoints</LI>
<LI>Verify that you can make connections between 2 local endpoints, a local
producer and a remote consumer, a remote producer and a local consumer, and two
2 remote endpoints. Test the local endpoints both registered and
unregistered.</LI>
<LI>2x Connect() on same consumer should give an error</LI>
<LI>The other applications should receive an mCON notification, and adjust
their own local rosters accordingly</LI>
<LI>If you are calling Connect() on a local producer, its Connected() hook
should be called. If you are calling Connect() on a remote producer, then its
own application should call the Connected() hook.</LI>
</UL><BR></LI>
<LI><P>Test the following for BMidiProducer::Disconnect():</P>
<UL>
<LI>Disconnect(NULL)</LI>
<LI>Disconnect(invalid consumer)</LI>
<LI>Disconnect() using an invalid producer</LI>
<LI>Send Mdis request with invalid IDs</LI>
<LI>Kill the midi_server just before you Disconnect()</LI>
<LI>Let the midi_server snooze, so the disconnect request times out</LI>
<LI>Have the midi_server return an error result code</LI>
<LI>On successful disconnect, verify that the consumer is removed from the
producer's list of endpoints</LI>
<LI>Verify that you can break connections between 2 local endpoints, a local
producer and a remote consumer, a remote producer and a local consumer, and two
2 remote endpoints. Test the local endpoints both registered and
unregistered.</LI>
<LI>Disconnecting 2 endpoints that were not connected should give an error</LI>
<LI>The other applications should receive an mDIS notification, and adjust
their own local rosters accordingly</LI>
<LI>If you are calling Disconnect() on a local producer, its Disconnected()
hook should be called. If you are calling Disconnect() on a remote producer,
then its own application should call the Disconnected() hook.</LI>
</UL><BR></LI>
<LI><P>Make a connection on a local producer. Release() the producer. The other
app should only receive an mDEL notification. Likewise if you have a connection
with a local consumer and you Release() that. However, now all apps should
throw away this consumer from the connection lists, invoking the Disconnected()
hook of local producers. The same thing happens if you Ctrl-C the app and
restart it. (Now the old endpoints are purged.)</P></LI>
<LI><P>BMidiProducer::IsConnected() should return false if you pass NULL or an
invalid consumer.</P></LI>
<LI><P>BMidiProducer::Connections() should return a new BList every time you
call it. The objects in this list are the BMidiConsumers that are connected to
this producer; verify that their reference counts are bumped for every call to
Connections().</P></LI>
</UL>
<HR SIZE="1">
<H2>Watching</H2>
<P><I>Required:</I> Client app that creates local consumer and producer
endpoints, and calls Register(), Unregister(), SetName(), SetLatency(), and
SetProperties(). It should also make and break connections.</P>
<UL>
<LI><P>When you call StartWatching(), you should receive B_MIDI_EVENT
notifications for all remote registered endpoints and the connections between
them. You will get no notifications for local endpoints, or for any connections
that involve unregistered endpoints. The BMidiRosterLooper should make a copy
of the BMessenger, so when the client destroys the original messenger, you will
still receive notifications. Verify that calling StartWatching() with the same
BMessenger twice in a row will also send the initial set of notifications
twice. StartWatching(NULL) should be ignored and does not remove the current
messenger.</P></LI>
<LI><P>Run the client app from two different Terminals. Verify that you receive
properly formatted B_MIDI_EVENT notifications when the other app changes the
attributes of its <I>registered</I> endpoints with the various Set() functions.
You should also receive notifications if the app Register()s or Unregister()s
its endpoints. That app that makes these changes does not receive the
notifications.</P></LI>
<LI><P>Run the client app from two different Terminals. Verify that you receive
properly formatted B_MIDI_EVENT notifications when the apps make and break
connections. Every app receives these connection notifications, whether the
endpoints are published or not. The app that makes and breaks the connections
does not receive any notifications.</P></LI>
<LI><P>StopWatching() should delete BMidiRosterLooper's BMessenger copy, if
any. Verify that you no longer receive B_MIDI_EVENT notifications for remote
endpoints after you have called StopWatching().</P></LI>
<LI><P>If the client is watching, and the BMidiRosterLooper receives an mDEL
notification for a registered remote endpoint, it should also send an
"unregistered" B_MIDI_EVENT to let the client know that this endpoint is no
longer available. If the endpoint was connected to anything, you'll also
receive "disconnected" B_MIDI_EVENTs.</P></LI>
<LI><P>If you get a "registered" event, and you do FindEndpoint() for that id,
you'll get its BMidiEndpoint object. If you get an "unregistered" event, then
FindEndpoint() returns NULL. So the events are send <I>after</I> the roster is
modified.</P></LI>
</UL>
<HR SIZE="1">
<H2>Event tests</H2>
<P><I>Required:</I> Several client apps that create and register consumer
endpoints that override the various MIDI event hook functions, as well as
producer endpoints that spray MIDI events. Also useful is a tool that lets you
make connections between all these endpoints (PatchBay), and a tool that lets
you monitor the MIDI events (MidiMonitor).</P>
<UL>
<LI><P>BMidiLocalProducer's spray functions should only try to send something
if there is one or more connected consumer. If the spray functions cannot
deliver their events, they simply ignore that consumer until the next spray.
(No connections are broken or anything.)</P></LI>
<LI><P>All spray functions except SprayData() should set the atomic flag to
true, even SpraySystemExclusive().</P></LI>
<LI><P>When you send a sysex message using SpraySystemExclusive(), it should
add 0xF0 in front of your data and 0xF7 at the back. When you call SprayData()
instead, no bytes are added to the MIDI event data.</P></LI>
<LI><P>Verify that all events arrive correctly and that the latency is minimal,
even when the load is heavy (i.e. many events are being sprayed to many
different consumers).</P></LI>
<LI><P>Verify that the BMidiLocalConsumer destructor properly destroys the
corresponding port and event thread before it returns.</P></LI>
<LI><P>BMidiLocalConsumer should ignore messages that are too small, addressed
to another consumer, or otherwise invalid.</P></LI>
<LI><P>BMidiLocalConsumer's Data() hook should ignore all non-atomic events.
The rest of the events, provided they contain the correct number of bytes for
that kind of event, are passed on to the other hooks.</P></LI>
<LI><P>Hook a producer up to a consumer and call all SprayXXX() functions with
a variety of arguments to make sure the correct hooks are being called with the
correct values. Call SprayData() and SpraySystemExclusive() with NULL data
and/or length 0.</P></LI>
<LI><P>Call GetProducerID() from one of BMidiLocalConsumer's hooks to verify
that this indeed returns the ID of the producer that sprayed the
event.</P></LI>
<LI><P>To test timeouts, first call SetTimeout(system_time() + 2000000), spray
an event to the consumer, and wait 2 seconds. The consumer's Timeout() hook
should now be called. Try again, but now spray multiple events to the consumer.
The Timeout() hook should still be called after 2 seconds, measured from the
moment the timeout was set. Replace the call to SetTimeout() with
SetTimeout(0). After spraying the first event, you should immediately get the
Timeout() signal, because the target time was set in the past. Verify that
calling SetTimeout() only takes effect after at least one new event has been
received.</P></LI>
</UL>
<HR SIZE="1">
<H2>Other tests</H2>
<UL>
<LI><P>Kill the server. Now run a client app. It should recognize that the
server isn't running, and return error codes on all operations. Also kill the
server while the test app is running. From then on, the client app will return
error codes on all operations. Also bring it back up again while the test app
is still running. Now the client app's request messages will be delivered to
the server again, but the server will ignore them, because our app did not
register with this new instance of the server.</P></LI>
<LI><P>Start the midi_server and several client apps. Use PatchBay to make and
break a whole bunch of connections. Quit PatchBay. Start it again. Now the same
connections should show up. Run similar tests with MidiKeyboard. Also install
VirtualMidi (and run the old midi_server for the time being) to get a whole
bunch of fake MIDI devices.</P></LI>
<LI><P><I>Regression bug:</I> After you quit one client app, another app fails
to send request to the midi_server.</P>
<P><I>Required:</I> Client app that creates a new endpoint and registers it. In
the app's destructor, it unregisters and releases the endpoint.</P>
<P><I>How to reproduce:</I> Run the app from two different Terminals. Ctrl-C
app1. Start app1 again. From the Deskbar quit both apps at the same time (that
is possible because app1 and app2 both have the same signature). When it tries
to send the Unregister() request to the midi_server, app2 gives the error
"Cannot send msg to server". The error code is "Bad Port ID", which means that
the reply port is dead. The Mdel message from Release() is sent without any
problems, however, because that expects no reply back. This is not the only way
to reproduce the problem, but it seems to be the most reliable one.</P>
<P>The reason this happens is because you kill app1. When app2 sends a
synchronous request to the midi_server, the server re-used that same message to
notify the other apps. (Because it already contained all the necessary fields.)
But app1 is dead, the notification fails, and this (probably) wipes out the
reply address in the message. I changed the midi_server to create new BMessages
for the notifications, and was no longer able to reproduce the
problem.</P></LI>
</UL>
</BODY>
</HTML>