[ML lab 09-1] Neural Net for XOR

[Logistic regression for XOR]

XOR 문제를 logistic regression으로 학습시킬 수 있을까?

import tensorflow.compat.v1 as tf
import numpy as np
tf.disable_v2_behavior()

x_data = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype = np.float32)
y_data = np.array([[0], [1], [1], [0]], dtype = np.float32)

X = tf.placeholder(tf.float32)
Y = tf.placeholder(tf.float32)
W = tf.Variable(tf.random_normal([2, 1]), name = 'weight')
b = tf.Variable(tf.random_normal([1]), name = 'bias')

# Hypothesis using sibmoid: tf.div(1., 1. + tf.exp(tf.matmul(X, W)))
hypothesis = tf.sigmoid(tf.matmul(X, W) + b)

# cost/loss function
cost = - tf.reduce_mean(Y * tf.log(hypothesis) + (1 - Y) * tf.log(1 - hypothesis))
train = tf.train.GradientDescentOptimizer(learning_rate = 0.1).minimize(cost)

# Accuracy computation
# True if hypothesis>0/5 else False
predicted = tf.cast(hypothesis > 0.5, dtype = tf.float32)
accuracy = tf.reduce_mean(tf.cast(tf.equal(predicted, Y), dtype = tf.float32))

# Launch graph
with tf.Session() as sess:
    # Initialize TensorFlow variables
    sess.run(tf.global_variables_initializer())
    
    for step in range(10001):
        sess.run(train, feed_dict = {X: x_data, Y: y_data})
        if step % 100 == 0:
            print(step, sess.run(cost, feed_dict = {X: x_data, Y: y_data}), sess.run(W))
            
    # Accuracy report
    h, c, a = sess.run([hypothesis, predicted, accuracy], feed_dict = {X: x_data, Y: y_data})
    print("\nHypothesis:\n", h, "\nCorrect:\n", c, "\nAccuracy: ", a)

[출력]

아무리 돌려도 정확도가 50% 이상 나오지 않는다.

---

[NN for XOR]

Neural Net을 통해 XOR 문제를 해결하다!

import tensorflow.compat.v1 as tf
import numpy as np
tf.disable_v2_behavior()

x_data = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype = np.float32)
y_data = np.array([[0], [1], [1], [0]], dtype = np.float32)

X = tf.placeholder(tf.float32)
Y = tf.placeholder(tf.float32)

W1 = tf.Variable(tf.random_normal([2, 2]), name = 'weight1') # 2는 임의로 선택
b1 = tf.Variable(tf.random_normal([2]), name = 'bias1')
layer1 = tf.sigmoid(tf.matmul(X, W1) + b1)

W2 = tf.Variable(tf.random_normal([2, 1]), name = 'weight2')
b2 = tf.Variable(tf.random_normal([1]), name = 'bias2')
hypothesis = tf.sigmoid(tf.matmul(layer1, W2) + b2) # layer1과 multiply

# cost/loss function
cost = - tf.reduce_mean(Y * tf.log(hypothesis) + (1 - Y) * tf.log(1 - hypothesis))
train = tf.train.GradientDescentOptimizer(learning_rate = 0.1).minimize(cost)

# Accuracy computation
# True if hypothesis>0/5 else False
predicted = tf.cast(hypothesis > 0.5, dtype = tf.float32)
accuracy = tf.reduce_mean(tf.cast(tf.equal(predicted, Y), dtype = tf.float32))

# Launch graph
with tf.Session() as sess:
    # Initialize TensorFlow variables
    sess.run(tf.global_variables_initializer())
    
    for step in range(10001):
        sess.run(train, feed_dict = {X: x_data, Y: y_data})
        if step % 100 == 0:
            print(step, sess.run(cost, feed_dict = {X: x_data, Y: y_data}), sess.run(W))
            
    # Accuracy report
    h, c, a = sess.run([hypothesis, predicted, accuracy], feed_dict = {X: x_data, Y: y_data})
    print("\nHypothesis:\n", h, "\nCorrect:\n", c, "\nAccuracy: ", a)

[출력]

NN을 통해 정확도 100%인 XOR 모델을 구현할 수 있다.

---

[Wide NN for XOR]

위 코드에서 입력값을 더욱 wide하게 10으로 변경해 학습시킨다.

W1 = tf.Variable(tf.random_normal([2, 10]), name = 'weight1') # 10으로 더 wide하게
b1 = tf.Variable(tf.random_normal([10]), name = 'bias1')
layer1 = tf.sigmoid(tf.matmul(X, W1) + b1)

W2 = tf.Variable(tf.random_normal([10, 1]), name = 'weight2')
b2 = tf.Variable(tf.random_normal([1]), name = 'bias2')
hypothesis = tf.sigmoid(tf.matmul(layer1, W2) + b2)

[출력]

hypothesis를 살펴보면, 작은 값은 더욱 작아졌고 큰 값은 더욱 커졌다. 모델이 더 잘 학습됐다.

---

[Deep NN for XOR]

위 코드에서 더욱 deep 하게 4 layer로 변경해 학습시킨다.

W1 = tf.Variable(tf.random_normal([2, 10]), name = 'weight1')
b1 = tf.Variable(tf.random_normal([10]), name = 'bias1')
layer1 = tf.sigmoid(tf.matmul(X, W1) + b1)

W2 = tf.Variable(tf.random_normal([10, 10]), name = 'weight2')
b2 = tf.Variable(tf.random_normal([10]), name = 'bias2')
layer2 = tf.sigmoid(tf.matmul(layer1, W2) + b2)

W3 = tf.Variable(tf.random_normal([10, 10]), name = 'weight3')
b3 = tf.Variable(tf.random_normal([10]), name = 'bias3')
layer3 = tf.sigmoid(tf.matmul(layer2, W3) + b3)

W4 = tf.Variable(tf.random_normal([10, 1]), name = 'weight4')
b4 = tf.Variable(tf.random_normal([1]), name = 'bias4')
hypothesis = tf.sigmoid(tf.matmul(layer3, W4) + b4)

[출력]

마찬가지로 더욱 정확한 모델이 구현됐다.