java: is adding methods in a subclass design poor?

Is adding methods in a subclass a poor design?

No, it is not, but your claim is incorrect:

As to access m3 we have to launch B and this requires the use of the operator instance:

In your example, you are calling m3 directly after instantiation, so the if condition is always true:

A a = new B(); 
if(a instanceof B) 
{ 
    B b = (B) a; b.m3(); 
}

It can be easily rewritten as:

B b = new B();
b.m3(); 
A a = b;